3
$\begingroup$

In an MLP with ReLU activation functions after each hidden layer (except the final),

Let's say the final layer should output positive and negative values.

With ReLU intermediary activations, this is still possible because the final layer, despite taking in positive inputs only, can combine them to be negative.

However, would using leaky ReLU allow faster convergence? Because you can pass in negative values as input to the final layer instead of waiting till the final layer to make things negative

$\endgroup$
1
$\begingroup$

In short, yes Leaky Relu helps in faster convergence if your output requires both positive and negative values. But the catch is that you need to tune the negative slope of Leaky Relu which is a hyperparameter to get better accuracy.

$\endgroup$
  • $\begingroup$ How are you so sure? What exactly do you mean by ' tuning the slope to get better accuracy'? $\endgroup$ – DuttaA Jan 27 at 10:21
  • $\begingroup$ By "tuning the negative slope" I mean that there is a parameter to Leaky Relu which has to be tuned for better results. $\endgroup$ – varsh Jan 28 at 10:22
  • $\begingroup$ I know that....it is for the first time I hearing tuning of parameters for better accuracy, generally loss is the parameter mentioned..Also you didn't answer my other question. $\endgroup$ – DuttaA Jan 28 at 10:43
1
$\begingroup$

The answer depends on a case to case basis. It may so happen that a dataset performs very well on ReLu but takes more number of iterations to converge on leaky ReLu's or PReLu's or vice-versa. There are 2 arguments to consider here:

  • ReLu is the most non-linear among all other type of ReLu's, and by this not so mathematical term I mean to say that it has the steepest drop in slope at 0, as compared to any other type of modified ReLu's.
  • ReLu's omit negative values which can be a significant problem with context to data normalisation. As this video (~10:00) from Stanford explains how data normalisation is necessary in context of signs of weight updates, so we can very roughly say any form of Leaky ReLU's somewhat normalise the data.

So theoretically speaking (might not be mathematically rigorous) iff all the inputs have a positive correlation to the output(input increases, output also increases), ReLu should work very well and converge faster. Whereas if there is negative correlation as well then Leaky ReLu's might work better.

The point is that unless someone gives definitive mathematical relations of what's going on inside a NN when it is being trained, its hard to tell which will work well and which will not except from intuition.

$\endgroup$
0
$\begingroup$

Both the output and the the gradient at the negative part of leaky ReLU is 100 times lower than at the positive part. I doubt that they have any significant impact on training direction and/or on the final output of a trained model unless the model is severely underfitting.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.