3
$\begingroup$

I'm struggling to understand the underlying mechanics of CNNs so any help is appreciated. I have a network with a ReLU activation function which does perform signifigantly better than one with sigmoid. This is expected as ReLU solves the vanishing gradient problem. However, my understanding was the reason we implement nonlinearities is to separate data which cannot be separated linearly. But if ReLU is linear for all values we care about it shouldn't work at all?

Unless, of course, neurons are defined for negative values but then my question becomes "why does ReLU solve the vanishing gradient problem at all?", since the derivative of ReLU for x<0 = 0

$\endgroup$
1
  • $\begingroup$ Your premise doesn't follow your conclusion. It seems you're implying that we use ReLU on the model's weights. We multiply the weights by the input which might not necessarily yield values [0, 1]. $\endgroup$
    – Daniel
    Feb 17, 2018 at 23:37

1 Answer 1

-1
$\begingroup$

You are right about the fact that we use nonlinearity for classes that cant be seperated by a straight line. If you think about curves,as we know from the calculus, you can approximate a nonlinear function with lines. If you have infinite amount of lines you can mimic exactly the same function. When you increase or decrease the weight of some neuron you essentially increase or decrease the length of the line it represents. And ReLU solves the vanishing gradient problem so they are superior to sigmoid in my opinion too.

$\endgroup$
4
  • $\begingroup$ Thanks for the answer! Quick follow up, I don't understand how it solves the vanishing gradient problem since there are still infinitly many points where the derivative of ReLU is zero. $\endgroup$ Jan 19, 2018 at 8:38
  • 1
    $\begingroup$ stats.stackexchange.com/q/176794 $\endgroup$
    – Ege Keyvan
    Jan 19, 2018 at 8:58
  • $\begingroup$ Ege is correct. The dying relu problem only occurs in very specific instances(dying relu is where a neuron always outputs the same value). For example, when using gradient descent and backpropagation the forward pass can clamp the weights to zero. Karpathy has some good info about that in his backprop walkthrough $\endgroup$ Apr 19, 2018 at 5:13
  • $\begingroup$ "When you increase or decrease the weight of some neuron you essentially increase or decrease the length of the line it represents": please, clarify. $\endgroup$ Aug 17, 2018 at 15:05

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .