2
$\begingroup$

The derivative of ReLU is 0 if its output is lower than 0 - $d ReLU(x)/dReLU$ is $0$ if $x < 0$. Let's denote some net's output by $Out$, so if this net's last layer is ReLU then we get that $dOut/dReLU$ is $0$ if $Out < 0$. Subsequently, for every parameter $p$ in the net we would get that $dOut/dp$ is $0$. Does that mean that for every sample $x$ such that $Out(x) < 0$ the net doesn't learn at all from that sample since the derivative for each parameter is $0$?

$\endgroup$
  • $\begingroup$ I have never seen the ReLU being used as the activation function of the last layer. $\endgroup$ – nbro Apr 27 at 23:33
1
$\begingroup$

There is no benefit to using ReLU as the output activation of a neural network. As you said, the network will ignore training labels below zero and it will train on labels above zero as if no output activation were present. However, the problem you're describing can also occur for individual units of hidden layers, where ReLU activations are common. This is known as the dead ReLU problem. In practice, it is rarely a problem but it can be avoided with smooth rectifiers like ELU and Swish. Another interesting ideas is CRelu, which concatenates both positive and negative parts of the pre-activation, resulting in twice as many outputs, half of which always receive a non-zero gradient.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.