# If neurons are only defined for values between 0 and 1, how does ReLU differ from the identity?

I'm struggling to understand the underlying mechanics of CNNs so any help is appreciated. I have a network with a ReLU activation function which does perform signifigantly better than one with sigmoid. This is expected as ReLU solves the vanishing gradient problem. However, my understanding was the reason we implement nonlinearities is to separate data which cannot be separated linearly. But if ReLU is linear for all values we care about it shouldn't work at all?

Unless, of course, neurons are defined for negative values but then my question becomes "why does ReLU solve the vanishing gradient problem at all?", since the derivative of ReLU for x<0 = 0

• Your premise doesn't follow your conclusion. It seems you're implying that we use ReLU on the model's weights. We multiply the weights by the input which might not necessarily yield values [0, 1]. Feb 17 '18 at 23:37