# What is the advantage of using cross entropy loss & softmax?

I am trying to do the standard MNIST dataset image recognition test with a standard feed forward NN, but my network failed pretty badly. Now I have debugged it quite a lot and found & fixed some errors, but I had a few more ideas. For one, I am using the sigmoid activation function and MSE as an error function, but the internet suggests that I should rather use softmax for the output layer, and cross entropy loss as an error function. Now I get that softmax is a nice activation function for this task, because you can treat the output as a propability vector. But, while being a nice thing to have, that's more of a convinience thing, isn't it? Easier to visualize?

But when I looked at what the derivative of softmax & CEL combined is (my plan was to compute that in one step and then treat the activation function of the last layer as linear, as not to apply the softmax derivative again), I found:

$$\frac{δE}{δi}$$ = $$t$$$$o$$

(With $$i$$ being the input of the last layer, $$t$$ the one hot target vector and $$o$$ the prediction vector).

That is the same as the MSE derivative. So what benefits does softmax + CEL actually have when propagating, if the gradients produced by them are exactly the same?

• Where did you find that derivative? Also, even if that's the correct derivative (I didn't think about it and I don't want to think about it now), note that, if $o$ is the prediction vector, then $t - o$ is different if $o$ is a probability vector or an unnormalised one.
– nbro
Oct 11 '20 at 9:29

That is not the derivative of the softmax function. $$t - o$$ is the combined derivative of the softmax function and cross entropy loss. Cross entropy loss is used to simplify the derivative of the softmax function. In the end, you do end up with a different gradients. It would be like if you ignored the sigmoid derivative when using MSE loss and the outputs are different. Using softmax and cross entropy loss has different uses and benefits compared to using sigmoid and MSE. It will help prevent gradient vanishing because the derivative of the sigmoid function only has a large value in a very small space of it. It is similar to using a different cross entropy loss where the combined derivative of the loss and sigmoid is $$t - o$$.
Regarding the supposed derivative of the cross-entropy loss function preceded by the softmax, even if that derivative is correct (I didn't think about it and I don't want to think about it now), note that then $$t - o$$ is different depending on whether $$o$$ is a probability vector or an unnormalized vector (which can take arbitrarily large numbers). If $$o$$ is a probability vector and $$t$$ a one-hot encoded vector (i.e. also a probability vector), then all numbers of $$t - o$$ will be in the range $$[-1, 1]$$. However, if $$o_i$$ can be arbitrarily large, e.g. $$o_i = 10$$, then $$t_i - o_i \in [-10, -9]$$. So, the propagated error would be different if $$o$$ was not a probability vector.