# How am I supposed to code equation 4.57 from the book "Machine Learning: An Algorithmic Perspective"?

Consider the equation 4.57 (p. 108) from section 4.6 of the Book Machine Learning: An Algorithmic Perspective, where the derivative of the softmax function is explained

$$\delta_o(\kappa) = (y_\kappa - t_\kappa)y_\kappa(\delta_{\kappa K} - y_K),$$

which is derived from equation 4.55 (p. 107)

$$y_{\kappa}(1 - y_{\kappa}),$$

which is to compute the diagonal of the Jacobian, and equation 4.56 (p. 107)

$$-y_{\kappa}y_K$$

In the book, it is not explained how they go from 4.55 and 4.56 to 4.57, it is just given, but I cannot follow how it is derived.

Moreover, in equation 4.57, the Kronecker's delta function is used, but how would one handle cases $$i=j$$, then we must have some for loop? Does having an $$i$$ and $$j$$ imply we need a nested for loop?

Also, I have tried to just compute the derivative of softmax according to the $$i=j$$ case only, and my model was faster (since we're not computing the jacobian) and accurate, but this assumes the error function is logarithmic, which I would like to code the general case.

• Let's first clarify that we are trying to differentiate the softmax function with respect to $h_K$. What is $h_K$? $h_\kappa$ is the output of neuron $\kappa$, but what is capital $K$?
– nbro
Commented Nov 4, 2020 at 0:25