Why use ReLU over Leaky ReLU?

Question

From my understanding a leaky ReLU attempts to address issues of vanishing gradients and nonzero-centeredness by keeping neurons that fire with a negative value alive.

With just this info to go off of, it would seem that the leaky ReLU is just an overall improvement to the standard ReLU, yet ReLU still seems to be the gold standard of activation functions. Why is that?

Does the additional sparsity outweigh the value gained from negative local gradients? If so, is tanh just too sparse?

Answer 1

Your understanding or Leaky ReLU is correct, and, yes, it has been proposed to mitigate the dying neurons issue in ReLU: when these are negative, they got zeroed.

Regarding the answer of @Regresslt:

• I would not say that leaky relu slows done training due to its computation overhead, since it's quite simple to implement. Instead, ELU (exponential linear unit) is quite expensive and that can increase training time.
• Also it's true that the slope parameter (usually called $\alpha$), if mistuned can lead to worst performance, but one can try the PReLU that learns it.

Does the additional sparsity outweigh the value gained from negative local gradients?

That's an interesting. I mean the dying ReLU phenomenon is usually observed on a fraction of the units, not the totality. In my personal experience I've noticed (dense) layers with ReLU to be sparse as much as $50$ to $90\%$. In general this causes an under utilization of the capacity of the model, which may lead to under-fitting. But from another perspective can be seen as implicit $l_1$-regularization applied on each layer with ReLU, that is optimized along the model which may lead to better generalization performance: simply because the model learned to achieve the right capacity by itself. Also, if such large sparsity is achieved the use of Dropout on top can be totally non-beneficial, because it would drop also the remaining "active" units/neurons.

Is tanh just too sparse?

First, $\tanh$ is not sparse since the output values can be either $-1$ or $1$ when saturation is reached, which is its main issue that shares with sigmoid (that instead can lead to sparsity.) When tanh or sigmoid saturate you get vanishing gradients, and the training stops. Instead, the sparsity induced by ReLU does not reduce the gradient, since it will only be multiplied by some zero elements (corresponding to the died units.): the gradient magnitude depends on the overall output of the layer. In practice (but I'd say it's hard to happen), only if all units of all layers are died your network won't learn anymore, and the gradient would vanish because each layers would only output zeroes.

To conclude, ReLU is usually the first activation to try - it seems to always work sufficiently well - but remember also to pair the activation fn with the right choice of weight initialization, in order to prevent issues (dying units, saturation, etc), speed-up convergence, and even increase performance.