Consider the equation 4.57 (p. 108) from section 4.6 of the Book Machine Learning: An Algorithmic Perspective, where the derivative of the softmax function is explained

$$\delta_o(\kappa) = (y_\kappa - t_\kappa)y_\kappa(\delta_{\kappa K} - y_K),$$

which is derived from equation 4.55 (p. 107)

$$y_{\kappa}(1 - y_{\kappa}),$$

which is to compute the diagonal of the Jacobian, and equation 4.56 (p. 107)


In the book, it is not explained how they go from 4.55 and 4.56 to 4.57, it is just given, but I cannot follow how it is derived.

Moreover, in equation 4.57, the Kronecker's delta function is used, but how would one handle cases $i=j$, then we must have some for loop? Does having an $i$ and $j$ imply we need a nested for loop?

Also, I have tried to just compute the derivative of softmax according to the $i=j$ case only, and my model was faster (since we're not computing the jacobian) and accurate, but this assumes the error function is logarithmic, which I would like to code the general case.

  • $\begingroup$ Let's first clarify that we are trying to differentiate the softmax function with respect to $h_K$. What is $h_K$? $h_\kappa$ is the output of neuron $\kappa$, but what is capital $K$? $\endgroup$
    – nbro
    Commented Nov 4, 2020 at 0:25


You must log in to answer this question.