1
$\begingroup$

I am implementing a feed-forward neural network with leaky ReLU activation functions and back-propagation from scratch. Now, I need to compute the partial derivatives, but I don't know what the derivative of the Leaky ReLU is.

Here is the C# code for the leaky RELU function which I got from this site:

private double leaky_relu(double x)
{
    if (x >= 0)
        return x;
    else
        return x / 20;
}
$\endgroup$
1
$\begingroup$

The ReLU function has a parameter that determines the slope of the function when $x < 0$. If you want that constant to be $1/20$, then the function that you have mentioned gets the required derivative.

| improve this answer | |
$\endgroup$
  • $\begingroup$ So, using that example, for $x < 0$, the derivative would be $x * 0.5$. Is that correct? $\endgroup$ – Mike AI Feb 25 '18 at 2:16
  • $\begingroup$ It would be $0.05$ because $1/20 = 0.05$. $\endgroup$ – Mohsin Bukhari Feb 25 '18 at 8:20
2
$\begingroup$

Derivative gives the rate of change in $y$ for a small change in $x$ or the slope of a function at point $x$.

In the above function,

y = x      for x >= 0,     i.e. y/x = 1
y = x/20   for x < 0,      i.e.  y/x = 1/20

The following function returns the derivative of leaky ReLU as explained

private double leaky_relu_derivative(double x)
{
    if (x >= 0)
        return 1;
    else
        return 1.0 / 20;
}
| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.