New answers tagged

0 votes

What is the derivative of the Leaky ReLU activation function?

In terms of the Heaviside step function $$H(x)=\frac{1+sign(x)}{2}$$ the leaky ReLU can be written as $$ (\alpha + (1-\alpha)H(x))x $$ so its derivative is $$ \alpha + (1-\alpha)H(x) $$ In your ...

Top 50 recent answers are included