Machine Learning books generally explains that the error calculated for a given sample $i$ is:

$e_i = y_i - \hat{y_i}$

Where $\hat{y}$ is the target output and $y$ is the actual output given by the network. So, a loss function $L$ is calculated:

$L = \frac{1}{2N}\sum^{N}_{i=1}(e_i)^2$

The above scenario is explained for a binary classification/regression problem. Now, let's assume a MLP network with $m$ neurons in the output layer for a multiclass classification problem (generally one neuron per class).

What changes in the equations above? Since we now have multiple outputs, both $e_i$ and $y_i$ should be a vector?


Assuming you're using softmax on the last layer for classification, it sounds like a simple application of cross entropy loss from here on out: https://datascience.stackexchange.com/questions/20296/cross-entropy-loss-explanation


| improve this answer | |
  • 2
    $\begingroup$ Rather than linking to another post (which can be lost, e.g. if the author of that post deletes it), please, explain (preferably, in your own words, i.e. without just copying and pasting) the concepts (in this case, the cross-entropy loss function). $\endgroup$ – nbro Oct 21 at 16:19

As you say, the outputs are modeled as a vector, each output in one vector component.

In regression problems:

The most common loss function, like in the scalar case, is the square error. Skipping constants, it is defined as:

$$E=\sum_i ||\mathbf{y_i}-\mathbf{\hat{y_i}}||^2 = \sum_i (\mathbf{y_i}-\mathbf{\hat{y_i}})(\mathbf{y_i}-\mathbf{\hat{y_i}})$$


  • $\mathbf{y_i}$ vector is expected value for sample $i$ (note I do not use the same naming convention than the question).
  • $\mathbf{\hat{y_i}}$ vector is network output for same sample
  • ||.|| is vector norm
  • product of vectors is scalar/inner product.

The derivative respect some NN parameter $w$ is:

$$\frac{\partial}{\partial w}E=\frac{\partial}{\partial w} \sum_i (\mathbf{y_i}-\mathbf{\hat{y_i}})(\mathbf{y_i}-\mathbf{\hat{y_i}}) = -2 \sum_i (\mathbf{y_i}-\mathbf{\hat{y_i}})\frac{\partial \mathbf{\hat{y_i}}}{\partial w} $$

being $\frac{\partial \mathbf{\hat{y_i}}}{\partial w}$ the term that backtracking algorithm evaluates.

Multi-class classification problems:

Two options appears as most usual ones:

  • a) optimize the square error of probabilities. Target vector will be of the form (0,...0,1,0,...0) while network output will be something as (0.2,0.1,0.8,0.4,...). This case can be solved like regression ones.
  • b) optimize entropy. In this case, an usual loss function is cross-entropy:

$$ E = - \sum_c p(c) log(\hat p(c)) \text{ [definition]} $$ $$ E = - \sum_i \sum_c p_i(c) log(\hat p_i(c)) = - \sum_i log(\hat p_i(c_i)) \text{ [average]} $$


  • $c$ is some class
  • $p(c)$ is probability of class $c$ in train dataset
  • $\hat p(c)$ is probability of class $c$ in network output
  • $i$ is sample number
  • $c_i$ is expected (correct) class of sample $i$
  • $p_i(c)$ is ground truth, usually taken as 1 if $c=c_i$, 0 otherwise as in latest expression.
  • $\hat p_i(c)$ is net output for sample $i$ and class $c$.
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.