# Does net with ReLU not learn when output < 0?

The derivative of ReLU is 0 if its output is lower than 0 - $$d ReLU(x)/dReLU$$ is $$0$$ if $$x < 0$$. Let's denote some net's output by $$Out$$, so if this net's last layer is ReLU then we get that $$dOut/dReLU$$ is $$0$$ if $$Out < 0$$. Subsequently, for every parameter $$p$$ in the net we would get that $$dOut/dp$$ is $$0$$. Does that mean that for every sample $$x$$ such that $$Out(x) < 0$$ the net doesn't learn at all from that sample since the derivative for each parameter is $$0$$?

• I have never seen the ReLU being used as the activation function of the last layer.
– nbro
Apr 27 '20 at 23:33