so I am writing my own neural network library for a class project and I got everything working for a simple 2-class test using the distance (L2) cost function. I wanted to get a similar result using softmax and crossentropy instead.

I did the derivation of the formulas as in this article, but I tried computing the derivative of the cost function w.r.t. its inputs and the Jacobian of the softmax outputs w.r.t. the inputs separately.

This is what I did:

So, let $Y$ be the matrix with rows equal to the expected class probabilities, $\hat{Y}$ the outputs of the softmax layer, $X$ the inputs of the softmax, all of these matrices of size $n\times c$ (samples times classes). The cost function $J(Y,\hat{Y}) = -sum(Y * \log(\hat{Y}))$ where * is element by element multiplication.

Then, if $E$ is the cost function, I first compute

$$\frac{\partial E}{\partial \hat{Y}_j^{(i)}} = -\frac{Y_j^{(i)}}{\hat{Y}_j^{(i)}},\:\: i=1,\cdots,n : j=1,\cdots,c$$

This is a $n\times c$ matrix. Then, being $S$ the softMax function I compute the Jacobian of the softmax at each row of $X$, so a total of $n$ matrices of $c\times c$ dimensions.

$$JS^{(i)} = \left(\frac{\partial \hat{Y}_j^{(i)}}{\partial X_k^{(i)}} \right)_{j,k=1,\cdots,c},\:\: i=1,\cdots,n$$

Finally to compute the partial derivative of the error with respect to the inputs of the softmax I multiply the rows of $\frac{\partial E}{\partial \hat{Y}}$ with each jacobian, to get a new $n\times c$ matrix.

This way I have everything needed to compute:

$$\frac{\partial E}{\partial X_j^{(i)}}= \sum_{k=1}^c \frac{\partial E}{\partial \hat{Y}_k^{(i)}}\frac{\partial \hat{Y}_k^{(i)}}{\partial X_j^{(i)}} ,\:\: i=1,\cdots,n : j=1,\cdots,c$$

Is this correct? Where is the problem if not so? Might the numerical errors make everything fail?

Thanks in advance.


1 Answer 1


I found the bug on my code, now everything works just fine, so I am fairly sure that the derivation of the formulas is on point. Optimisation wise, clearly using the short formula on the end of the article is better and less prone to errors than using the full derivation.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .