# How to choose an activation function?

I choose the activation function for the output layer depending on the output that I need and the properties of the activation function that I know. For example, I choose the sigmoid function when I'm dealing with probabilities, a ReLU when I'm dealing with positive values, and a linear function when I'm dealing with general values.

In hidden layers, I use a leaky ReLU to avoid dead neurons instead of the ReLU and the tanh instead of the sigmoid. Of course, I don't use a linear function in hidden units.

However, the choice for them in the hidden layer is mostly due to trial and error.

Is there any rule of thumb of which activation function is likely to work good in some situations? Take the term situations as general as possible: it could be referring to the depth of the layer, to the depth of the NN, to the number of neurons for that layer, to the optimizer that we chose, to the number of input features of that layer, to the application of this NN, etc.

In his/her answer, cantordust refers to other activation functions that I didn't mention, like ELU and SELU. This infos are more than welcomed. However, the more activation functions I discover the more I'm confused in the choice of the function to use in hidden layers. And I don't think that flipping a coin is a good way of choosing an activation function.

It seems to me that you already understand the shortcomings of ReLUs and sigmoids (like dead neurons in the case of plain ReLU). I would suggest looking at ELU (exponential linear units) and SELU (self-normalising version of ELU). Under some mild assumptions, the latter have the nice property of self-normalisation, which mitigates the problem of vanishing and exploding gradients. In addition, they propagate normalisation - i.e., they guarantee that the input to the next layer will have zero mean and unit variance.

Edit:

It would be incredibly difficult to recommend an activation function that works for all use cases (although I believe that SELU was designed so that it would do the right thing with pretty much any input). There are many considerations - how difficult it is to compute the derivative (if it is differentiable at all!), how quickly a network with your chosen AF converges, how smooth it is, whether it satisfies the conditions of the universal approximation theorem, whether it preserves normalisation, and so on. You may or may not care about some or any of those.

The bottom line is that there is no universal rule for choosing an activation function for hidden layers. Personally, I like to use sigmoids (especially tanh) because they are nicely bounded and very fast to compute, but most importantly because they work for my use cases. Others recommend leaky ReLU for the input and hidden layers as a go-to function if your network fails to learn. You can even mix and match activation functions to evolve neural nets for fancy applications.

At the end of the day, you are probably going to get as many opinions as there are people about the right choice of activation function, so the short answer should probably be: start with the AF of the day (leaky ReLU / SELU?) and work your way through other AFs in order of decreasing popularity if your network struggles to learn anything.

• Right, I forgot about the propagation of normalization. Thanks for reminding me. However, the question is still unanswered. There is a rule or something to choose which activation functions should I put in the hidden layers? For simplicity I'm talking only about simply fully connected layers. I don't want to complicate the topic with convolutions, pooling, etc. – gvgramazio Jul 9 '18 at 12:32
• @gvgramazio I edited the answer, hopefully it is a bit more useful now. – cantordust Jul 9 '18 at 14:40
• With the edit I think that you answered to my question as well as provided interested links (in particular the one about mix and match). Unfortunately is not the answer that I wanted to hear. I'll keep my question open for a little more days. If nobody comes up with a better answer I will mark yours as accepted. – gvgramazio Jul 9 '18 at 16:15

I don't know what kind of neural networks you are working on. But one should also consider tanh activation functions when dealing with recurrent neural network. The why is to avoid exploding gradient issues since the tanh function is bounded at the difference of the RELU function for instance.

• In the question I stated that I use tanh and sigmoid, not only ReLU. Also, to keep it simple I'm referring in general to classical hidden fully connected layers. If you think that the fact that we are dealing with a recurrent neural network is significant for the choice of the activation function please state the reason for that. The exploding/vanishing phenomena could happens in non recurrent neural network as well. – gvgramazio Jul 9 '18 at 16:11