# What is the derivative of the Leaky ReLU activation function?

I am implementing a feed-forward neural network with leaky ReLU activation functions and back-propagation from scratch. Now, I need to compute the partial derivatives, but I don't know what the derivative of the Leaky ReLU is.

Here is the C# code for the leaky RELU function which I got from this site:

private double leaky_relu(double x)
{
if (x >= 0)
return x;
else
return x / 20;
}

• – nbro
May 30, 2020 at 12:54

The ReLU function has a parameter that determines the slope of the function when $$x < 0$$. If you want that constant to be $$1/20$$, then the function that you have mentioned gets the required derivative.

• So, using that example, for $x < 0$, the derivative would be $x * 0.5$. Is that correct? Feb 25, 2018 at 2:16
• It would be $0.05$ because $1/20 = 0.05$. Feb 25, 2018 at 8:20

Derivative gives the rate of change in $$y$$ for a small change in $$x$$ or the slope of a function at point $$x$$.

In the above function,

y = x      for x >= 0,     i.e. y/x = 1
y = x/20   for x < 0,      i.e.  y/x = 1/20


The following function returns the derivative of leaky ReLU as explained

private double leaky_relu_derivative(double x)
{
if (x >= 0)
return 1;
else
return 1.0 / 20;
}


In terms of the Heaviside step function $$H(x)=\frac{1+sign(x)}{2}$$

the leaky ReLU can be written as $$(\alpha + (1-\alpha)H(x))x$$

so its derivative is

$$\alpha + (1-\alpha)H(x)$$

In your example, $$\alpha = \frac{1}{20}$$