What is the mathematical definition of an activation function to be used in a neural network?

So far I did not find a precise one, summarizing which criterions (e.g. monotonicity, differentiability, etc.) are required. Any recommendations for literature about this or – even better – the definition itself?

In particular, one major point which is unclear for me is differentiability. In lots of articles, this is required for the activation function, but then, out of nowhere, ReLU (which is not differentiable) is used. I totally understand why we need to be able to take derivatives of it and I also understand why we can use ReLU in practice anyway, but how does one formalize this?

  • $\begingroup$ I think activation function's requirements are coupled with what loss function (which might be dictated by type of problem you are trying to solve) and method of optimization you are using. For example genetic algos don't require many criterion's as compared traditional ML losses, but still most widely use MSE loss as a measure of convergence, but they can use any other measure too. $\endgroup$ – DuttaA Jan 10 at 10:33
  • $\begingroup$ Relu is differentiable at all places except exactly 0. This doesn't pose much of a problem in practice because the input being exactly 0 is quite uncommon. However, for an activation function that is monotonic, and differentiable at all places, see here: fadev.ai/wp-content/uploads/2019/05/sigmoid.png $\endgroup$ – Recessive Jan 10 at 10:58

There is no strict definition of suitability of an activation function for neural networks. Instead there are a number of desirable traits, and functions that don't meet them or come close enough may perform badly in general (but those functions may still work in specific cases)

If you are using gradient descent as a training method, then differentiability is closest to being a requirement. However, even then as you noticed with ReLU, it is not an absolute requirement, provided behaviour close to discontinuities is good. For example $\frac{1}{x}$ or $log{x}$ make bad choices here due to how they behave for values near to $0$.

Wikipedia summarises some important desirable traits, plus compares popular activation functions in this table.

Nonlinear – When the activation function is non-linear, then a two-layer neural network can be proven to be a universal function approximator. The identity activation function does not satisfy this property. When multiple layers use the identity activation function, the entire network is equivalent to a single-layer model.

Range – When the range of the activation function is finite, gradient-based training methods tend to be more stable, because pattern presentations significantly affect only limited weights. When the range is infinite, training is generally more efficient because pattern presentations significantly affect most of the weights. In the latter case, smaller learning rates are typically necessary.

Continuously differentiable – This property is desirable (RELU is not continuously differentiable and has some issues with gradient-based optimization, but it is still possible) for enabling gradient-based optimization methods. The binary step activation function is not differentiable at 0, and it differentiates to 0 for all other values, so gradient-based methods can make no progress with it.

Monotonic – When the activation function is monotonic, the error surface associated with a single-layer model is guaranteed to be convex.

Smooth functions with a monotonic derivative – These have been shown to generalize better in some cases.

Approximates identity near the origin – When activation functions have this property, the neural network will learn efficiently when its weights are initialized with small random values. When the activation function does not approximate identity near the origin, special care must be used when initializing the weights.

You can see that ReLU is an outlier on more than one of these traits. The reason that it is popular is that despite these weaknesses, the things it does well - speed of operation plus helping combat the vanishing gradient problem - make it a solid choice for very deep networks. In fact the success of ReLU has inspired use of a number of similar-looking activation functions that hope to keep its benefits whilst having more of these desirable traits.

Ultmately though, almost any non-linear function can be used successfully in a neural network. The more of the desirable traits it has, the more likely you can use it in general cases with existing approaches to training.

| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.