The reason for using ReLU is to somehow make the output positive(and add non-linearity). But why do we need to have ReLU in the last layers when we do not have any convolution to get negative values? (Assume VGG). after the last convolution layer, we apply ReLU and from that point, there are no more convolutions, therefore, it is not possible to get negative values. So, there is no need to make already positive values positive. I cannot understand the reason. I think it would be because of backpropagation and we do not want to make the vector (in the flatter layer) zero as the derivative of the constant is zero. Is it the case? Can anyone give me an explanation? I have read this question and this question. But I am not sure whether the answers are the only explanations for using ReLU in the last layers.

  • 1
    $\begingroup$ "from that point there are no more convolution and therefore, it is not possible to get negative values", why is this the case? If you have a FC layer, you will have something like $x_1 w_1 + ... x_n w_n$, here $w_i$ could be negative, so this sum could still be negative. $\endgroup$
    – nbro
    Feb 23 at 23:01


You must log in to answer this question.

Browse other questions tagged .