# Can anyone show me the derivative of Leaky RELU in C#?

I am in the process of getting back into AI programming after some time out and have been building my neural net in C#.NET. I managed to get all of the feed-forward stuff working very eloquently but I am not using Sigmoid as the activation function; instead I am using Leaky RELU as I heard it is best for deep learning.

I began working through the back-propagation material today and ran into a road-block as I need to know the derivative of Leaky RELU in order to calculate the changes in weights. My calculus is lacking so I was hoping someone might be able to help.

Here is the code for the leaky RELU function which I got from https://www.codeproject.com/Articles/1220276/ReInventing-Neural-Networks:

private double ReLU(double x)
{
if (x >= 0)
return x;
else
return x / 20;
}


-Mike

The relu function has a parameter which determines the slope of the function when x < 0. If you want that constant to be 1/20 then the function that you have mentioned gets the required derivative.

• So using that example, for x < 0 the derivative would be x * 0.5 is that correct? – Mike AI Feb 25 '18 at 2:16
• it would be 0.05 because 1/20 = 0.05 – Mohsin Bukhari Feb 25 '18 at 8:20
• That's what I meant - sorry! But yes, thank you for answering and for clarifying. I really appreciate it. – Mike AI Feb 26 '18 at 4:03
• It is my pleasure to help. Let me know if you have any other questions. – Mohsin Bukhari Feb 26 '18 at 16:22

Derivative gives the rate of change in y for a small change in x or the slope[ y/x ] of a function at point x. In the above function,

           y = x     for x >= 0,    i.e. y/x = 1
x/20  for x < 0      i.e  y/x = 1/20


the following function returns the derivative of leaky ReLU as explained

private double ReLU_prime(double x)
{
if (x >= 0)
return 1;
else
return 1.0 / 20;
}


hope this helps.