Does leaky relu help learning if the final output needs negative values?

Question

In an MLP with ReLU activation functions after each hidden layer (except the final),

Let's say the final layer should output positive and negative values.

With ReLU intermediary activations, this is still possible because the final layer, despite taking in positive inputs only, can combine them to be negative.

However, would using leaky ReLU allow faster convergence? Because you can pass in negative values as input to the final layer instead of waiting till the final layer to make things negative

Answer 1

In short, yes Leaky Relu helps in faster convergence if your output requires both positive and negative values. But the catch is that you need to tune the negative slope of Leaky Relu which is a hyperparameter to get better accuracy.

Answer 2

The answer depends on a case to case basis. It may so happen that a dataset performs very well on ReLu but takes more number of iterations to converge on leaky ReLu's or PReLu's or vice-versa. There are 2 arguments to consider here:

• ReLu is the most non-linear among all other type of ReLu's, and by this not so mathematical term I mean to say that it has the steepest drop in slope at 0, as compared to any other type of modified ReLu's.
• ReLu's omit negative values which can be a significant problem with context to data normalisation. As this video (~10:00) from Stanford explains how data normalisation is necessary in context of signs of weight updates, so we can very roughly say any form of Leaky ReLU's somewhat normalise the data.

So theoretically speaking (might not be mathematically rigorous) iff all the inputs have a positive correlation to the output(input increases, output also increases), ReLu should work very well and converge faster. Whereas if there is negative correlation as well then Leaky ReLu's might work better.

The point is that unless someone gives definitive mathematical relations of what's going on inside a NN when it is being trained, its hard to tell which will work well and which will not except from intuition.

Answer 3

Both the output and the the gradient at the negative part of leaky ReLU is 100 times lower than at the positive part. I doubt that they have any significant impact on training direction and/or on the final output of a trained model unless the model is severely underfitting.