I saw this implementation of backpropagation in MATLAB, where the loss function used is MSE, and the last layer's activation function was sigmoid.

enter image description here

I denoted the portions of the formula for what I thought they represented. .* represents an element wise multiplication in MATLAB. I tried to expand this for other cases, where I used Softmax as the last layer's activation, and categorical cross entropy for the loss. Since the implementation above requires the two parts to be the same shape, and the backwards pass of Softmax gives the 2D Jacobian for a 1D Vector, how would I element-wise multiply this with the loss function's derivative, another Vector? Would taking the dot product give me the result I need?

  • $\begingroup$ The Jacobian for the softmax should be a $m \times n$ matrix, where $m$ is the number of parameters and $n$ is the number of output neurons (for example, if you have 10 classes, $n = 10$). HOWEVER, I wouldn't really try to extend that Matlab formula to the softmax + cross-entropy case. Instead, I would only ask how to apply the chain rule of calculus to compute the gradient of the cross-entropy when the activation function of the last layer is the softmax. $\endgroup$ – nbro Jan 6 at 22:19
  • $\begingroup$ So, I think that your main first problem/question is: "How to compute the gradient of the cross-entropy loss function with respect to the parameters of the neural network in the case of multi-class classification problems with a softmax activation function in the last layer?". So, I don't understand why you're trying to extend, in an ad-hoc manner, other formulas that are applicable for other cases to this case. You don't need to compute the Jacobian (although it may pop up in the calculations), but you need to compute the gradient of the cross-entropy. $\endgroup$ – nbro Jan 6 at 22:25
  • $\begingroup$ So, what I suggest is that you do not start by trying to extend other formulas, such as the one you show, that are applicable for other cases to this case softmax + cross-entropy, but that you ask just the question that I've written above in bold, and maybe also ask the question "How does the Jacobian of the softmax come into play when computing the gradient?", which is also one thing that seems to confuse you. $\endgroup$ – nbro Jan 6 at 22:27
  • $\begingroup$ So, just to be clear softmax is not the same thing as the cross-entropy. The cross-entropy is a function of the softmax (usually). That's why I suggest that you ask the 2 questions in bold above, though they may actually require some effort. $\endgroup$ – nbro Jan 6 at 22:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.