# How does backpropagation with unbounded activation functions such as ReLU work?

I am in the process of writing my own basic machine learning library in Python as an exercise to gain a good conceptual understanding. I have successfully implemented backpropagation for activation functions such as $$\tanh$$ and the sigmoid function. However, these are normalised in their outputs. A function like ReLU is unbounded so its outputs can blow up really fast. In my understanding, a classification layer, usually using the SoftMax function, is added at the end to squash the outputs between 0 and 1.

How does backpropagation work with this? Do I just treat the SoftMax function as another activation function and compute its gradient? If so, what is that gradient and how would I implement it? If not, how does the training process work? If possible, a pseudocode answer is preferred.

Backprop through ReLU is easier than backprop through sigmoid activations. For positive activations, you just pass through the input gradients as they were. For negative activations you just set the gradients to 0.

Regarding softmax, the easiest approach is to consider it a part of the negative log-likelihood loss. In other words, I am suggesting to directly derive gradients of that loss with respect to the softmax input. The result is very elegant and extremely easy to implement. Try to derive that yourself!

• Thanks so much! I went through the calculus and I got the SoftMax minus 1, is that correct? Also for implementation, do I simply use this gradient in backpropagation and feed it back through the network like normal? – Archie Shahidullah Feb 17 at 8:10
• You are on the right path! Note however that the negative log-likelihood depends on correct labels, while there are no labels in your expression for the NLL-softmax gradient. We have a contradiction: a supervised learning which does not use supervision. Try to correct that! – ssegvic Feb 17 at 17:36