How to choose an activation function for the hidden layers?

Question

I choose the activation function for the output layer depending on the output that I need and the properties of the activation function that I know. For example, I choose the sigmoid function when I'm dealing with probabilities, a ReLU when I'm dealing with positive values, and a linear function when I'm dealing with general values.

In hidden layers, I use a leaky ReLU to avoid dead neurons instead of the ReLU, and the tanh instead of the sigmoid. Of course, I don't use a linear function in hidden units.

However, the choice for them in the hidden layer is mostly due to trial and error.

Is there any rule of thumb of which activation function is likely to work well in some situations?

Take the term situations as general as possible: it could be referring to the depth of the layer, to the depth of the NN, to the number of neurons for that layer, to the optimizer that we chose, to the number of input features of that layer, to the application of this NN, etc.

The more activation functions I discover the more I'm confused in the choice of the function to use in hidden layers. I don't think that flipping a coin is a good way of choosing an activation function.

Answer 1

It seems to me that you already understand the shortcomings of ReLUs and sigmoids (like dead neurons in the case of plain ReLU).

You may want to look at ELU (exponential linear units) and SELU (self-normalising version of ELU). Under some mild assumptions, the latter has the nice property of self-normalisation, which mitigates the problem of vanishing and exploding gradients. In addition, they propagate normalisation - i.e., they guarantee that the input to the next layer will have zero mean and unit variance.

However, it would be incredibly difficult to recommend an activation function (AF) that works for all use cases, although I believe that SELU was designed so that it would do the right thing with pretty much any input.

There are many considerations - how difficult it is to compute the derivative (if it is differentiable at all!), how quickly a NN with your chosen AF converges, how smooth it is, whether it satisfies the conditions of the universal approximation theorem, whether it preserves normalisation, and so on. You may or may not care about some or any of those.

The bottom line is that there is no universal rule for choosing an activation function for hidden layers. Personally, I like to use sigmoids (especially tanh) because they are nicely bounded and very fast to compute, but most importantly because they work for my use cases. Others recommend leaky ReLU for the input and hidden layers as a go-to function if your NN fails to learn. You can even mix and match activation functions to evolve NNs for fancy applications.

At the end of the day, you are probably going to get as many opinions as there are people about the right choice of activation function, so the short answer should probably be: start with the AF of the day (leaky ReLU / SELU?) and work your way through other AFs in order of decreasing popularity if your NN struggles to learn anything.

Answer 2

***Take my answer as a side note to that given by cantordust:

If one can verify that an activation function perform well in some cases, that good behavior often extrapolates to other problems. Thus, by testing activation functions on a few different problems, one can often infer how well (or badly) it will perform on most problems. The following video shows how different activation functions perform in different problems:

https://www.youtube.com/watch?v=Hb3vIYUQ_I8

One can verify that an activation function usually perform well in all cases, or the other way around: it does it poorly in all cases. As cantordust says, I would recommend always starting with leaky ReLU: it is simple, efficient, and generally produces nice results in a wide variety of problems. It also evades the dying ReLU problem, and does not suffer from the vanishing gradient problem. The only thing to keep in mind is the exploding gradient problem if the neural network is too deep, or if it is a recurrent neural network, which are essentially the same concept.

The video shows that other activation functions worth trying (in addition to leaky ReLU) are Gaussian, Sinusoid, or Tanh.

Answer 3

I don't know what kind of neural networks you are working on. But one should also consider tanh activation functions when dealing with recurrent neural network. The why is to avoid exploding gradient issues since the tanh function is bounded at the difference of the RELU function for instance.