1
$\begingroup$

I am implementing a feed-forward neural network with leaky ReLU activation functions and back-propagation from scratch. Now, I need to compute the partial derivatives, but I don't know what the derivative of the Leaky ReLU is.

Here is the C# code for the leaky RELU function which I got from this site:

private double leaky_relu(double x)
{
    if (x >= 0)
        return x;
    else
        return x / 20;
}
$\endgroup$
1

2 Answers 2

1
$\begingroup$

The ReLU function has a parameter that determines the slope of the function when $x < 0$. If you want that constant to be $1/20$, then the function that you have mentioned gets the required derivative.

$\endgroup$
2
  • $\begingroup$ So, using that example, for $x < 0$, the derivative would be $x * 0.5$. Is that correct? $\endgroup$
    – Mike AI
    Feb 25, 2018 at 2:16
  • $\begingroup$ It would be $0.05$ because $1/20 = 0.05$. $\endgroup$ Feb 25, 2018 at 8:20
2
$\begingroup$

Derivative gives the rate of change in $y$ for a small change in $x$ or the slope of a function at point $x$.

In the above function,

y = x      for x >= 0,     i.e. y/x = 1
y = x/20   for x < 0,      i.e.  y/x = 1/20

The following function returns the derivative of leaky ReLU as explained

private double leaky_relu_derivative(double x)
{
    if (x >= 0)
        return 1;
    else
        return 1.0 / 20;
}
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .