0
$\begingroup$

I am in the process of getting back into AI programming after some time out and have been building my neural net in C#.NET. I managed to get all of the feed-forward stuff working very eloquently but I am not using Sigmoid as the activation function; instead I am using Leaky RELU as I heard it is best for deep learning.

I began working through the back-propagation material today and ran into a road-block as I need to know the derivative of Leaky RELU in order to calculate the changes in weights. My calculus is lacking so I was hoping someone might be able to help.

Here is the code for the leaky RELU function which I got from https://www.codeproject.com/Articles/1220276/ReInventing-Neural-Networks:

private double ReLU(double x)
{
    if (x >= 0)
        return x;
    else
        return x / 20;
}

Thanks in advance.

-Mike

$\endgroup$
1
$\begingroup$

The relu function has a parameter which determines the slope of the function when x < 0. If you want that constant to be 1/20 then the function that you have mentioned gets the required derivative.

$\endgroup$
  • $\begingroup$ So using that example, for x < 0 the derivative would be x * 0.5 is that correct? $\endgroup$ – Mike AI Feb 25 '18 at 2:16
  • $\begingroup$ it would be 0.05 because 1/20 = 0.05 $\endgroup$ – Mohsin Bukhari Feb 25 '18 at 8:20
  • $\begingroup$ That's what I meant - sorry! But yes, thank you for answering and for clarifying. I really appreciate it. $\endgroup$ – Mike AI Feb 26 '18 at 4:03
  • $\begingroup$ It is my pleasure to help. Let me know if you have any other questions. $\endgroup$ – Mohsin Bukhari Feb 26 '18 at 16:22
2
$\begingroup$

Derivative gives the rate of change in y for a small change in x or the slope[ y/x ] of a function at point x. In the above function,

           y = x     for x >= 0,    i.e. y/x = 1
               x/20  for x < 0      i.e  y/x = 1/20

the following function returns the derivative of leaky ReLU as explained

private double ReLU_prime(double x)
{
    if (x >= 0)
        return 1;
    else
        return 1.0 / 20;
}

hope this helps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.