# How to choose an activation function for the hidden layers?

I choose the activation function for the output layer depending on the output that I need and the properties of the activation function that I know. For example, I choose the sigmoid function when I'm dealing with probabilities, a ReLU when I'm dealing with positive values, and a linear function when I'm dealing with general values.

In hidden layers, I use a leaky ReLU to avoid dead neurons instead of the ReLU, and the tanh instead of the sigmoid. Of course, I don't use a linear function in hidden units.

However, the choice for them in the hidden layer is mostly due to trial and error.

Is there any rule of thumb of which activation function is likely to work well in some situations?

Take the term situations as general as possible: it could be referring to the depth of the layer, to the depth of the NN, to the number of neurons for that layer, to the optimizer that we chose, to the number of input features of that layer, to the application of this NN, etc.

The more activation functions I discover the more I'm confused in the choice of the function to use in hidden layers. I don't think that flipping a coin is a good way of choosing an activation function.

It seems to me that you already understand the shortcomings of ReLUs and sigmoids (like dead neurons in the case of plain ReLU).

You may want to look at ELU (exponential linear units) and SELU (self-normalising version of ELU). Under some mild assumptions, the latter has the nice property of self-normalisation, which mitigates the problem of vanishing and exploding gradients. In addition, they propagate normalisation - i.e., they guarantee that the input to the next layer will have zero mean and unit variance.

However, it would be incredibly difficult to recommend an activation function (AF) that works for all use cases, although I believe that SELU was designed so that it would do the right thing with pretty much any input.

There are many considerations - how difficult it is to compute the derivative (if it is differentiable at all!), how quickly a NN with your chosen AF converges, how smooth it is, whether it satisfies the conditions of the universal approximation theorem, whether it preserves normalisation, and so on. You may or may not care about some or any of those.

The bottom line is that there is no universal rule for choosing an activation function for hidden layers. Personally, I like to use sigmoids (especially tanh) because they are nicely bounded and very fast to compute, but most importantly because they work for my use cases. Others recommend leaky ReLU for the input and hidden layers as a go-to function if your NN fails to learn. You can even mix and match activation functions to evolve NNs for fancy applications.

At the end of the day, you are probably going to get as many opinions as there are people about the right choice of activation function, so the short answer should probably be: start with the AF of the day (leaky ReLU / SELU?) and work your way through other AFs in order of decreasing popularity if your NN struggles to learn anything.

• If my input is a length 4 vector containing only positive number, and my output is also a length 4 vector containing only positive number. Then, ReLU, leaky ReLU would always be a good choice, am I correct? not the tanh, sigmoid, softmax, etc. – FantasticAI Jan 31 at 3:56

***Take my answer as a side note to that given by cantordust:

If one can verify that an activation function perform well in some cases, that good behavior often extrapolates to other problems. Thus, by testing activation functions on a few different problems, one can often infer how well (or badly) it will perform on most problems. The following video shows how different activation functions perform in different problems: