Which functions can be activation functions?

Question

What are the required characteristics of an activation function (in a neural network)? Which functions can be activation functions?

For example, which of the functions below can be used as an activation function?

$$f(x) = \frac{2}{\pi} \tan^{-1}(x)$$

which looks like

or

$$f(x) = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{x} e^{-\frac{t^2}{2}} dt$$

which looks

Answer 1

The main characteristic of an activation function is to bring a non linearity property into the NN. For the hidden layer, there is no need for the function to be bounded. The last layer should use a function which range correspond to what you want.

For regression, you usually re-scale your output data to [-1,1] or [0,1] and you use a tanh or sigmoid function in the last layer

For classification, you want to obtain probabilities : use a softmax function in the last layer.

For the hidden layers some functions are better than others :

• The gradient should be fast to compute (from the perspective of your computer).
• If you use too much layer you will have the vanishing gradient problem if the derivative of your activation is too close to zero. You need a large zone of the domain with a derivative not close to zero.

In practice the Relu function defined by f(x)=max(0, x) works very well and is very simple.