I am trying to do the standard MNIST dataset image recognition test with a standard feed forward NN, but my network failed pretty badly. Now I have debugged it quite a lot and found & fixed some errors, but I had a few more ideas. For one, I am using the sigmoid activation function and MSE as an error function, but the internet suggests that I should rather use softmax for the output layer, and cross entropy loss as an error function. Now I get that softmax is a nice activation function for this task, because you can treat the output as a propability vector. But, while being a nice thing to have, that's more of a convinience thing, isn't it? Easier to visualize?

But when I looked at what the derivative of softmax & CEL combined is (my plan was to compute that in one step and then treat the activation function of the last layer as linear, as not to apply the softmax derivative again), I found:

$\frac{δE}{δi}$ = $t$$o$

(With $i$ being the input of the last layer, $t$ the one hot target vector and $o$ the prediction vector).

That is the same as the MSE derivative. So what benefits does softmax + CEL actually have when propagating, if the gradients produced by them are exactly the same?

  • $\begingroup$ Where did you find that derivative? Also, even if that's the correct derivative (I didn't think about it and I don't want to think about it now), note that, if $o$ is the prediction vector, then $t - o$ is different if $o$ is a probability vector or an unnormalised one. $\endgroup$
    – nbro
    Oct 11, 2020 at 9:29

2 Answers 2


Short answer: larger gradients

That is not the derivative of the softmax function. $t - o$ is the combined derivative of the softmax function and cross entropy loss. Cross entropy loss is used to simplify the derivative of the softmax function. In the end, you do end up with a different gradients. It would be like if you ignored the sigmoid derivative when using MSE loss and the outputs are different. Using softmax and cross entropy loss has different uses and benefits compared to using sigmoid and MSE. It will help prevent gradient vanishing because the derivative of the sigmoid function only has a large value in a very small space of it. It is similar to using a different cross entropy loss where the combined derivative of the loss and sigmoid is $t - o$.

Information on derivatives of cross entropy with sigmoid function and with softmax function. I would also suggest some more research on cross entropy loss functions beyond my links.

  • $\begingroup$ so cross entropy makes loss convergence faster, compared to mse? $\endgroup$
    – Dan D.
    Mar 11, 2021 at 2:36
  • 1
    $\begingroup$ @datdinhqouc When the last layer is the sigmoid function then cross entropy should make loss convergence faster and prevent vanishing gradients. $\endgroup$
    – S2673
    Mar 14, 2021 at 19:26

If you look at the definition of the cross-entropy (e.g. here), you will see that it is defined for probability distributions (in fact, it comes from information theory). You can also show that the maximization of the (binomial/Bernoulli) log-likelihood is equivalent to the minimization of the cross-entropy, i.e. when you minimize the cross-entropy you actually maximize the log-likelihood of the parameters given your labelled data. Hence the use of the softmax is theoretically founded.

Regarding the supposed derivative of the cross-entropy loss function preceded by the softmax, even if that derivative is correct (I didn't think about it and I don't want to think about it now), note that then $t - o$ is different depending on whether $o$ is a probability vector or an unnormalized vector (which can take arbitrarily large numbers). If $o$ is a probability vector and $t$ a one-hot encoded vector (i.e. also a probability vector), then all numbers of $t - o$ will be in the range $[-1, 1]$. However, if $o_i$ can be arbitrarily large, e.g. $o_i = 10$, then $t_i - o_i \in [-10, -9]$. So, the propagated error would be different if $o$ was not a probability vector.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .