What is the derivative of the Leaky ReLU activation function?

Question

I am implementing a feed-forward neural network with leaky ReLU activation functions and back-propagation from scratch. Now, I need to compute the partial derivatives, but I don't know what the derivative of the Leaky ReLU is.

Here is the C# code for the leaky RELU function which I got from this site:

private double leaky_relu(double x)
{
    if (x >= 0)
        return x;
    else
        return x / 20;
}

Answer 1

The ReLU function has a parameter that determines the slope of the function when $x < 0$. If you want that constant to be $1/20$, then the function that you have mentioned gets the required derivative.

Answer 2

Derivative gives the rate of change in $y$ for a small change in $x$ or the slope of a function at point $x$.

In the above function,

y = x      for x >= 0,     i.e. y/x = 1
y = x/20   for x < 0,      i.e.  y/x = 1/20

The following function returns the derivative of leaky ReLU as explained

private double leaky_relu_derivative(double x)
{
    if (x >= 0)
        return 1;
    else
        return 1.0 / 20;
}

Answer 3

In terms of the Heaviside step function $$H(x)=\frac{1+sign(x)}{2}$$

the leaky ReLU can be written as $$ (\alpha + (1-\alpha)H(x))x $$

so its derivative is

$$ \alpha + (1-\alpha)H(x) $$

In your example, $\alpha = \frac{1}{20}$