3
$\begingroup$

I am very new to machine learning. I am following the course offered by Andrew Ng. I am very confused about how we train our neural network for multi-class classification.

Let's say we have $K$ classes. For $K$ classes, we will be training $K$ different neural networks.

But do we train one neural network at a time for all features, or do we train all $K$ neural networks at a time for one feature?

Please, explain the complete procedures.

$\endgroup$

2 Answers 2

2
$\begingroup$

Let's say we have $K$ classes. For $K$ classes, we will be training $K$ different neural networks.

No, you still train one network.

With binary classification tasks, where you have only two mutually exclusive categories, like "yes/no" or "true/false", you can get away with a single output node with a sigmoid activation. The output of the sigmoid is interpreted as indicating one category for values $> 0.5$ and the other for values $\leq 0.5$.

With multi-class classification, you have $K$ outputs (one for each category). The problem, in this case, is that if the network gets the class wrong, in general, you cannot decide in one step which one of the other $K - 1$ categories is the correct one. So, the output is actually passed through an extra softmax layer, which outputs probabilities for each class.

But do we train one neural network at a time for all features, or do we train all $K$ neural networks at a time for one feature?

You present all features for each training example to the network at the same time. So, for $N$ features you have $N$ input nodes, and you feed all of them into the neural network.

$\endgroup$
5
  • $\begingroup$ okay it means for K classes and M training examples we will be training our neural network K*M times. $\endgroup$ Jul 8, 2018 at 12:01
  • $\begingroup$ One more doubt sir.Do we apply gradient descent(any varient) when using neural network? in this code they are applying advance optimization before Back Propagation. According to me in BP as we are already finding the best set of weights then applying gradient descent is irrelevant in this case.Am I correct or missing something ? $\endgroup$ Jul 8, 2018 at 12:08
  • $\begingroup$ @ReenaKandari If you have M data points, you would be using a subset (generally, between 60% and 80% of M) for training, and you do that multiple times (= epochs). However, you don't have to train the network K times for each training example - the whole point of softmax is that the correct class is "promoted" while all others are "suppressed" simultaneously, if that makes sense. $\endgroup$
    – cantordust
    Jul 9, 2018 at 22:32
  • $\begingroup$ @ReenaKandari Re your other question: yes, in the overwhelming majority of cases neural networks are optimised using gradient descent. I am not proficient in Matlab, but it seems to me that in the example you provided they are minimising a cost function using gradient descent. $\endgroup$
    – cantordust
    Jul 9, 2018 at 22:33
  • $\begingroup$ sir @cantordust, actually I am not aware of how Softmax works and how classes are 'promoted' and 'suppressed simultaneously '.Can you please explain it a little bit more that how these things works in terms of neural network? $\endgroup$ Jul 14, 2018 at 2:32
2
$\begingroup$

Let us suppose that you are training a neural network for classfing images of vehicles , then the input vector , image of the "vehicle" will be a 2D array of pixels. This undergoes several transformations at each layer of the neural network , the last layer of the neural network produces another vector whose dimensions are lesser than the original image vector.

So the network is mapping images to some vectors in a high dimensional space. In order to classify the images it is now sufficient to classify the vectors obtained from the network of their corresponding images, you can do this with a simple "linear" classifier using a softmax layer.

So all the layers of the network except the last layer are transforming the image representation to a "vector". This vector is classified by a linear softmax classifier by the last layer of neural network.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .