Which functions can be activation functions?

Question

What are the required characteristics of an activation function (in a neural network)? Which functions can be activation functions?

For example, which of the functions below can be used as an activation function?

$$f(x) = \frac{2}{\pi} \tan^{-1}(x)$$

which looks like

or

$$f(x) = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{x} e^{-\frac{t^2}{2}} dt$$

which looks

Answer 1

The main characteristic of an activation function is to bring a non-linearity property into the neural network. For the hidden layer, there is no need for the function to be bounded. The last layer should use a function whose range corresponds to what you want.

For regression, you usually re-scale your output data to $[-1,1]$ or $[0,1]$ and you use a tanh (hyperbolic tangent) or sigmoid function in the last layer

For classification, you want to obtain probabilities: use a softmax function in the last layer.

For the hidden layers some functions are better than others :

• The gradient should be fast to compute (from the perspective of your computer).

• If you use too many hidden layers, you will have the vanishing gradient problem if the derivative of your activation is too close to zero. You need a large zone of the domain with a derivative not close to zero.

In practice, the ReLU function defined as $f(x)=\max(0, x)$ works very well and is very simple.