Consider the equation 4.57 (p. 108) from section 4.6 of the Book Machine Learning: An Algorithmic Perspective, where the derivative of the softmax function is explained

$$\delta_o(\kappa) = (y_\kappa - t_\kappa)y_\kappa(\delta_{\kappa K} - y_K),$$

which is derived from equation 4.55 (p. 107)

$$y_{\kappa}(1 - y_{\kappa}),$$

which is to compute the diagonal of the Jacobian, and equation 4.56 (p. 107)


In the book, it is not explained how they go from 4.55 and 4.56 to 4.57, it is just given, but I cannot follow how it is derived.

Moreover, in equation 4.57, the Kronecker's delta function is used, but how would one handle cases $i=j$, then we must have some for loop? Does having an $i$ and $j$ imply we need a nested for loop?

Also, I have tried to just compute the derivative of softmax according to the $i=j$ case only, and my model was faster (since we're not computing the jacobian) and accurate, but this assumes the error function is logarithmic, which I would like to code the general case.

  • $\begingroup$ Let's first clarify that we are trying to differentiate the softmax function with respect to $h_K$. What is $h_K$? $h_\kappa$ is the output of neuron $\kappa$, but what is capital $K$? $\endgroup$
    – nbro
    Nov 4 '20 at 0:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.