Since the neural network nonlinearities allow for nonlinear transformations that can stretch and squish the function, how can the ReLU activation function do this?

I think for the sigmoid nonlinearity (1/1+e^-x) or tanh nonlinearity (e^x-e^-x/e^x+e^-x), it makes sense how it can because there is an exponentiation in the function so the function can get squashed and curved. (Is it learning polynomial terms?)

But since Relu is max(0, x), It is sets some values to 0 otherwise does nothing. It seems that it's just mapping a lot of values to 0, which makes it easier to fit a line to separate the output categories? Is this the wrong way to interpret nonlinearities?



You must log in to answer this question.