3
$\begingroup$

I am in the process of writing my own basic machine learning library in Python as an exercise to gain a good conceptual understanding. I have successfully implemented backpropagation for activation functions such as $\tanh$ and the sigmoid function. However, these are normalised in their outputs. A function like ReLU is unbounded so its outputs can blow up really fast. In my understanding, a classification layer, usually using the SoftMax function, is added at the end to squash the outputs between 0 and 1.

How does backpropagation work with this? Do I just treat the SoftMax function as another activation function and compute its gradient? If so, what is that gradient and how would I implement it? If not, how does the training process work? If possible, a pseudocode answer is preferred.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

Backprop through ReLU is easier than backprop through sigmoid activations. For positive activations, you just pass through the input gradients as they were. For negative activations you just set the gradients to 0.

Regarding softmax, the easiest approach is to consider it a part of the negative log-likelihood loss. In other words, I am suggesting to directly derive gradients of that loss with respect to the softmax input. The result is very elegant and extremely easy to implement. Try to derive that yourself!

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thanks so much! I went through the calculus and I got the SoftMax minus 1, is that correct? Also for implementation, do I simply use this gradient in backpropagation and feed it back through the network like normal? $\endgroup$
    – Archie Shahidullah
    Feb 17, 2019 at 8:10
  • $\begingroup$ You are on the right path! Note however that the negative log-likelihood depends on correct labels, while there are no labels in your expression for the NLL-softmax gradient. We have a contradiction: a supervised learning which does not use supervision. Try to correct that! $\endgroup$
    – ssegvic
    Feb 17, 2019 at 17:36
  • $\begingroup$ You say "For positive activations, you just pass through the input gradients as they were. For negative activations you just set the gradients to 0.", but you don't explain why that must be the case, from a calculus perspective. I think this answer would improve if you explain that more in detail. $\endgroup$
    – nbro
    Nov 12, 2020 at 12:46
  • $\begingroup$ @nbro I assume that if OP is able to write a backprop for tanh, then he should know what is the derivative of ReLU. $\endgroup$
    – ssegvic
    Nov 13, 2020 at 13:34
  • $\begingroup$ @ssegvic How does the derivative of ReLU come from the derivative of tanh? It's just a convention that the derivative of ReLU at zero is 0, given that the left and right derivatives at zero are different. You don't explain this in your answer. $\endgroup$
    – nbro
    Nov 13, 2020 at 13:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .