Why do activation functions need to be differentiable in the context of neural networks?

Question

Why should an activation function of a neural network be differentiable? Is it strictly necessary or is it just advantageous?

Answer 1

No, it is not necessary that an activation function is differentiable. In fact, one of the most popular activation functions, the rectifier, is non-differentiable at zero!

This can create problems with learning, as numerical gradients calculated near a non-differentiable point can be incorrect. The "kinks" section in these lecture notes discuss the issue.

I don't know the full answer as to whether this is a problem in practice and how people get around them. There are ways to avoid getting these incorrect gradients for some non-differentiable functions, as the lecture notes I discussed above mention.

Answer 2

Training

While "running" a neural network can be done with any activation functions, we usually want to train it - i.e., adjust its parameters so that the result would be closer to what we desire.

This is commonly done by backpropagation and variations of gradient descent, which requires the existence of a gradient - i.e., requires activation function to be differentiable. The adjustment of each parameter is calculated from the gradient of the activation function(s) that this parameter affects, so if you cannot get a gradient, then this approach can't be used.

Answer 3

Lets go through your question point by point.

• Should an activation function be differentiable?

No, it is not of any compulsion of it to be differentiable. We use Re-Lu's which have a non-differentiable point at 0. But this is simply a trivial case, since the point 0 will never be reached unless we run out of precision points to denote extemely less numbers.

So lets take another example, the Perceptron learning algorithm. In this algorithm, there is no satisfactory way to evaluate the performance of a particular solution. So we don't have a cost function. But still we are able to reach to a solution, albeit maybe not the best. I'll come to the point why it is not used later.

NN's can be broadly thought of as just function approximators. Normally, you give it some continuous function, the NN adjusts it by elongating, shifting, distorting parts of that function by changing only and only the parameters of the function and not the nature of the function itself i.e. it'll decompose into the same sort of a Fourier series as before, with only phase and amplitude differences. You can also design a NN on lines of a random search puzzle. You give each node of a, say single hidden layer NN a part of the function to be approximated i.e. between some intervals -a < x < b say. Say the connection weights to each output layer is fixed. You can only change the part of the function (i.e.parts broken like a jigsaw from a function to be approximated) you want to give to a node, from your fixed box of parts. So you see, in this case the NN will try almost all combinations of giving different parts to different nodes until the function is perfectly approximated. So you see, this can also be a type of learning for NN without any continuous function.

• Why should an activation function of a neural network be differentiable?

The main advantages of differentiable function is the mathematics behind it. You can easily handle huge amounts of data just by simple mathematics. 1000's of years of mathematical theories can be applied to verify the working of your NN, predict how will it work, select best algorithms. In discrete non-differentiable activation functions you have no realistic way of predicting what will be the final result.

Say in a problem like this, if you apply the perceptron learning algo., do you have any realistic way to predict where the final decision boundary will rest? Whereas, if you train it by continuous methods, you can easily plot the cost function versus weights, check the global minima, get the weights which will be the final weights. So we easily solved it by plotting (definite matrix methods also exist).

• Is it advantageous to have a differentiable activation function? Why?

So why is it advantegeous?

--> Final answers easily predictable (if relatively small data set).

--> Easy to draw graphs and visualize the working of your NN and adjust your hyper-parameters accordingly.

--> Easy to apply time tested mathematical tools to test/evaluate the effectiveness of your algorithm.

--> No sudden changes in error and so your weights will also not change suddenly due to stray readings.

These are all the advantages I could think of and I am sue there are more. All these tools are not available for discrete algorithms, if you think (can't give the full intuition - too lengthy).

I gave some more intuition here:

What is the purpose of an activation function in Neural Networks?

Feel free to add anything I missed by editing the answer.

Hope this helps!

Answer 4

As already said , Activation function is almost differentiable in every neural net to facillitate Training as well as to calculate tendency towards a certain result when some parameter is changed. But I just wanted to point out that The Output function need not be differentiable in all cases. We can have non-sigmoid (hard-limiter threshold)output nodes but still train them with backpropagation and gradient descent.