3
$\begingroup$

I'm struggling to understand the underlying mechanics of CNNs so any help is appreciated. I have a network with a ReLU activation function which does perform signifigantly better than one with sigmoid. This is expected as ReLU solves the vanishing gradient problem. However, my understanding was the reason we implement nonlinearities is to separate data which cannot be separated linearly. But if ReLU is linear for all values we care about it shouldn't work at all?

Unless, of course, neurons are defined for negative values but then my question becomes "why does ReLU solve the vanishing gradient problem at all?", since the derivative of ReLU for x<0 = 0

Improve this question
$\endgroup$
1
  • $\begingroup$ Your premise doesn't follow your conclusion. It seems you're implying that we use ReLU on the model's weights. We multiply the weights by the input which might not necessarily yield values [0, 1]. $\endgroup$
    – Daniel
    Feb 17 '18 at 23:37
-1
$\begingroup$

You are right about the fact that we use nonlinearity for classes that cant be seperated by a straight line. If you think about curves,as we know from the calculus, you can approximate a nonlinear function with lines. If you have infinite amount of lines you can mimic exactly the same function. When you increase or decrease the weight of some neuron you essentially increase or decrease the length of the line it represents. And ReLU solves the vanishing gradient problem so they are superior to sigmoid in my opinion too.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks for the answer! Quick follow up, I don't understand how it solves the vanishing gradient problem since there are still infinitly many points where the derivative of ReLU is zero. $\endgroup$
    – Emil Wormbs
    Jan 19 '18 at 8:38
  • 1
    $\begingroup$ stats.stackexchange.com/q/176794 $\endgroup$
    – Ege Keyvan
    Jan 19 '18 at 8:58
  • $\begingroup$ Ege is correct. The dying relu problem only occurs in very specific instances(dying relu is where a neuron always outputs the same value). For example, when using gradient descent and backpropagation the forward pass can clamp the weights to zero. Karpathy has some good info about that in his backprop walkthrough $\endgroup$
    – hisairnessag3
    Apr 19 '18 at 5:13
  • $\begingroup$ "When you increase or decrease the weight of some neuron you essentially increase or decrease the length of the line it represents": please, clarify. $\endgroup$
    – pasaba por aqui
    Aug 17 '18 at 15:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.