As you say, the outputs are modeled as a vector, each output in one vector component.
In regression problems:
The most common loss function, like in the scalar case, is the square error. Skipping constants, it is defined as:
$$E=\sum_i ||\mathbf{y_i}-\mathbf{\hat{y_i}}||^2 = \sum_i (\mathbf{y_i}-\mathbf{\hat{y_i}})(\mathbf{y_i}-\mathbf{\hat{y_i}})$$
where:
- $\mathbf{y_i}$ vector is expected value for sample $i$ (note I do not use the same naming convention than the question).
- $\mathbf{\hat{y_i}}$ vector is network output for same sample
- ||.|| is vector norm
- product of vectors is scalar/inner product.
The derivative respect some NN parameter $w$ is:
$$\frac{\partial}{\partial w}E=\frac{\partial}{\partial w} \sum_i (\mathbf{y_i}-\mathbf{\hat{y_i}})(\mathbf{y_i}-\mathbf{\hat{y_i}}) = -2 \sum_i (\mathbf{y_i}-\mathbf{\hat{y_i}})\frac{\partial \mathbf{\hat{y_i}}}{\partial w} $$
being $\frac{\partial \mathbf{\hat{y_i}}}{\partial w}$ the term that backtracking algorithm evaluates.
Multi-class classification problems:
Two options appears as most usual ones:
- a) optimize the square error of probabilities. Target vector will be of the form (0,...0,1,0,...0) while network output will be something as (0.2,0.1,0.8,0.4,...). This case can be solved like regression ones.
- b) optimize entropy. In this case, an usual loss function is cross-entropy:
$$ E = - \sum_c p(c) log(\hat p(c)) \text{ [definition]} $$
$$ E = - \sum_i \sum_c p_i(c) log(\hat p_i(c)) = - \sum_i log(\hat p_i(c_i)) \text{ [average]} $$
where:
- $c$ is some class
- $p(c)$ is probability of class $c$ in train dataset
- $\hat p(c)$ is probability of class $c$ in network output
- $i$ is sample number
- $c_i$ is expected (correct) class of sample $i$
- $p_i(c)$ is ground truth, usually taken as 1 if $c=c_i$, 0 otherwise as in latest expression.
- $\hat p_i(c)$ is net output for sample $i$ and class $c$.