1
$\begingroup$

Consider the equation 4.57 (p. 108) from section 4.6 of the Book Machine Learning: An Algorithmic Perspective, where the derivative of the softmax function is explained

$$\delta_o(\kappa) = (y_\kappa - t_\kappa)y_\kappa(\delta_{\kappa K} - y_K),$$

which is derived from equation 4.55 (p. 107)

$$y_{\kappa}(1 - y_{\kappa}),$$

which is to compute the diagonal of the Jacobian, and equation 4.56 (p. 107)

$$-y_{\kappa}y_K$$

In the book, it is not explained how they go from 4.55 and 4.56 to 4.57, it is just given, but I cannot follow how it is derived.

Moreover, in equation 4.57, the Kronecker's delta function is used, but how would one handle cases $i=j$, then we must have some for loop? Does having an $i$ and $j$ imply we need a nested for loop?

Also, I have tried to just compute the derivative of softmax according to the $i=j$ case only, and my model was faster (since we're not computing the jacobian) and accurate, but this assumes the error function is logarithmic, which I would like to code the general case.

$\endgroup$
1
  • $\begingroup$ Let's first clarify that we are trying to differentiate the softmax function with respect to $h_K$. What is $h_K$? $h_\kappa$ is the output of neuron $\kappa$, but what is capital $K$? $\endgroup$
    – nbro
    Nov 4 '20 at 0:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.