What are the required characteristics of an activation function (in a neural network)? Which functions can be activation functions?

For example, which of the functions below can be used as an activation function?

$$f(x) = \frac{2}{\pi} \tan^{-1}(x)$$

which looks like

enter image description here


$$f(x) = \frac{1}{\sqrt{2\pi}} \int_{-\infty}^{x} e^{-\frac{t^2}{2}} dt$$

which looks

enter image description here

  • 2
    $\begingroup$ the derivative of the activation function should be easy to compute , so the first one is better than the second. $\endgroup$ – riemann77 Jun 5 '18 at 11:06

The main characteristic of an activation function is to bring a non linearity property into the NN. For the hidden layer, there is no need for the function to be bounded. The last layer should use a function which range correspond to what you want.

For regression, you usually re-scale your output data to [-1,1] or [0,1] and you use a tanh or sigmoid function in the last layer

For classification, you want to obtain probabilities : use a softmax function in the last layer.

For the hidden layers some functions are better than others :

  • The gradient should be fast to compute (from the perspective of your computer).
  • If you use too much layer you will have the vanishing gradient problem if the derivative of your activation is too close to zero. You need a large zone of the domain with a derivative not close to zero.

In practice the Relu function defined by f(x)=max(0, x) works very well and is very simple.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.