I'm struggling to understand the underlying mechanics of CNNs so any help is appreciated. I have a network with a ReLU activation function which does perform signifigantly better than one with sigmoid. This is expected as ReLU solves the vanishing gradient problem. However, my understanding was the reason we implement nonlinearities is to separate data which cannot be separated linearly. But if ReLU is linear for all values we care about it shouldn't work at all?
Unless, of course, neurons are defined for negative values but then my question becomes "why does ReLU solve the vanishing gradient problem at all?", since the derivative of ReLU for x<0 = 0