How is the error calculated with multiple output neurons in the neural network?

Question

Machine Learning books generally explains that the error calculated for a given sample $i$ is:

$e_i = y_i - \hat{y_i}$

Where $\hat{y}$ is the target output and $y$ is the actual output given by the network. So, a loss function $L$ is calculated:

$L = \frac{1}{2N}\sum^{N}_{i=1}(e_i)^2$

The above scenario is explained for a binary classification/regression problem. Now, let's assume a MLP network with $m$ neurons in the output layer for a multiclass classification problem (generally one neuron per class).

What changes in the equations above? Since we now have multiple outputs, both $e_i$ and $y_i$ should be a vector?

Answer 1

Assuming you're using softmax on the last layer for classification, it sounds like a simple application of cross entropy loss from here on out: https://datascience.stackexchange.com/questions/20296/cross-entropy-loss-explanation

Edit:

Answer 2

As you say, the outputs are modeled as a vector, each output in one vector component.

In regression problems:

The most common loss function, like in the scalar case, is the square error. Skipping constants, it is defined as:

$$E=\sum_i ||\mathbf{y_i}-\mathbf{\hat{y_i}}||^2 = \sum_i (\mathbf{y_i}-\mathbf{\hat{y_i}})(\mathbf{y_i}-\mathbf{\hat{y_i}})$$

where:

• $\mathbf{y_i}$ vector is expected value for sample $i$ (note I do not use the same naming convention than the question).
• $\mathbf{\hat{y_i}}$ vector is network output for same sample
• ||.|| is vector norm
• product of vectors is scalar/inner product.

The derivative respect some NN parameter $w$ is:

$$\frac{\partial}{\partial w}E=\frac{\partial}{\partial w} \sum_i (\mathbf{y_i}-\mathbf{\hat{y_i}})(\mathbf{y_i}-\mathbf{\hat{y_i}}) = -2 \sum_i (\mathbf{y_i}-\mathbf{\hat{y_i}})\frac{\partial \mathbf{\hat{y_i}}}{\partial w} $$

being $\frac{\partial \mathbf{\hat{y_i}}}{\partial w}$ the term that backtracking algorithm evaluates.

Multi-class classification problems:

Two options appears as most usual ones:

• a) optimize the square error of probabilities. Target vector will be of the form (0,...0,1,0,...0) while network output will be something as (0.2,0.1,0.8,0.4,...). This case can be solved like regression ones.
• b) optimize entropy. In this case, an usual loss function is cross-entropy:

$$ E = - \sum_c p(c) log(\hat p(c)) \text{ [definition]} $$ $$ E = - \sum_i \sum_c p_i(c) log(\hat p_i(c)) = - \sum_i log(\hat p_i(c_i)) \text{ [average]} $$

where:

• $c$ is some class
• $p(c)$ is probability of class $c$ in train dataset
• $\hat p(c)$ is probability of class $c$ in network output
• $i$ is sample number
• $c_i$ is expected (correct) class of sample $i$
• $p_i(c)$ is ground truth, usually taken as 1 if $c=c_i$, 0 otherwise as in latest expression.
• $\hat p_i(c)$ is net output for sample $i$ and class $c$.