From my understanding a leaky ReLU attempts to address issues of vanishing gradients and nonzero-centeredness by keeping neurons that fire with a negative value alive.

With just this info to go off of, it would seem that the leaky ReLU is just an overall improvement to the standard ReLU, yet ReLU still seems to be the gold standard of activation functions. Why is that?

Does the additional sparsity outweigh the value gained from negative local gradients? If so, is tanh just too sparse?

  • $\begingroup$ Can I ask what you mean by sparsity? $\endgroup$ Nov 12, 2023 at 17:53
  • $\begingroup$ @JobHunter69 A sparse activation function is more strict when it comes to letting non-zero activations through than a non-sparse function. This is beneficial b/c it leads to a more generalizable model that "focuses" on only the most distinguishing features of the data. Read Deep Sparse Rectifier Neural Networks by Bengio et al. for a more thorough explanation of the topic. $\endgroup$
    – John Brown
    Nov 14, 2023 at 19:31

2 Answers 2


Your understanding or Leaky ReLU is correct, and, yes, it has been proposed to mitigate the dying neurons issue in ReLU: when these are negative, they got zeroed.

Regarding the answer of @Regresslt:

  • I would not say that leaky relu slows done training due to its computation overhead, since it's quite simple to implement. Instead, ELU (exponential linear unit) is quite expensive and that can increase training time.
  • Also it's true that the slope parameter (usually called $\alpha$), if mistuned can lead to worst performance, but one can try the PReLU that learns it.

Does the additional sparsity outweigh the value gained from negative local gradients?

That's an interesting. I mean the dying ReLU phenomenon is usually observed on a fraction of the units, not the totality. In my personal experience I've noticed (dense) layers with ReLU to be sparse as much as $50$ to $90\%$. In general this causes an under utilization of the capacity of the model, which may lead to under-fitting. But from another perspective can be seen as implicit $l_1$-regularization applied on each layer with ReLU, that is optimized along the model which may lead to better generalization performance: simply because the model learned to achieve the right capacity by itself. Also, if such large sparsity is achieved the use of Dropout on top can be totally non-beneficial, because it would drop also the remaining "active" units/neurons.

Is tanh just too sparse?

First, $\tanh$ is not sparse since the output values can be either $-1$ or $1$ when saturation is reached, which is its main issue that shares with sigmoid (that instead can lead to sparsity.) When tanh or sigmoid saturate you get vanishing gradients, and the training stops. Instead, the sparsity induced by ReLU does not reduce the gradient, since it will only be multiplied by some zero elements (corresponding to the died units.): the gradient magnitude depends on the overall output of the layer. In practice (but I'd say it's hard to happen), only if all units of all layers are died your network won't learn anymore, and the gradient would vanish because each layers would only output zeroes.

To conclude, ReLU is usually the first activation to try - it seems to always work sufficiently well - but remember also to pair the activation fn with the right choice of weight initialization, in order to prevent issues (dying units, saturation, etc), speed-up convergence, and even increase performance.


Leaky ReLU is indeed an improvement over the standard ReLU activation function, but comes with some of the following limitations:

  • It may suffer from the "dying ReLU" problem, where a large fraction of units can become inactive and never recover. This can happen when the input to a unit is always negative, causing the slope of the activation function to be zero.
  • The parameter that controls the slope of the function can be difficult to tune, which can lead to suboptimal performance.
  • Leaky ReLU can also be less computationally efficient because it requires additional computation to calculate the slope of the function for non-positive inputs. This extra computation can slow down the training process, especially when dealing with large datasets.

On the other hand, ReLU is a popular activation function due to its simplicity and effectiveness in many deep learning applications which is maybe why it's considered the "gold standard". It is also relatively computationally efficient and helps to speed up training.

Regarding the sparsity versus negative local gradients trade-off, this can depend on the specific problem being solved and the architecture being used. Tanh can be more prone to vanishing gradients, but it can also provide smoother gradients and be more robust to noisy inputs. Ultimately, the choice of activation function should be based on empirical testing and optimization for the specific task at hand.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .