# Why to prefer ReLU over Linear activation functions?

ReLU : y = max(0,x)

Linear : y = x

The ReLU nonlinearity just clips the values < 0 to 0 and passes everything else; then why not to use a linear activation function instead as it will pass all the gradient information during backpropagation.

I do see that Parametric ReLU (PReLU) does provide this possiblity.

I just want to know if there is a proper explanation to using ReLU as default or it is just based on observations that it performs better on the training sets.

• Use as interpretation of the activation function result the one of "error". Relu has a value 0, meaning expected optimal result is reached. Lineal has a minimum of less infinite, meaning the error can be less than zero (not applicable in most cases) and always posible to improve (near than never applicable) – pasaba por aqui May 19 '18 at 18:00
• This question should be migrated to: datascience.stackexchange.com. – JahKnows Jun 1 '18 at 8:47
• A simple intuition behind this, is that an ANN with all linear activations is analogous to linear regression – hisairnessag3 Feb 18 '19 at 10:30

The ReLu is a non-linear activation function. Check out this question for the intuition behind using ReLu's (also check out the comments). There is a very simple reason of why we do not use a linear activation function.

Say you have a feature vector $x_0$ and weight vector $W_1$. Passing through a layer in a Neural Net will give the output as

$W_1^T * x_0 = x_1$

(dot product of weights and input vector). Now passing the output through next layer will give you

$W_2^T * x_1 = x_2$

So expanding this we get

$x_2 = W_2^T * W_1^T * x_0 = W_2^T * W_1^T * x_0 = W_{compact}^T * x_0$

Thus as you can see there is a linear relationship between input and output, and the function we want to model is generally non-linear, and so we cannot model it.

You can check out my answer here on non-linear activation.

Parametric ReLu has few advantages over normal ReLu. Here is a great answer by @NeilSlater on the same. It is basically trying to tell us that if we use ReLu's we will end up with a lot of redundant or dead nodes in a Neural Net (those which have a negative output) which do not contribute to the result, and thus do not have a derivative. Thus to approximate a function we will require a larger NN, whereas parametric ReLu's absolve us of this problem,(thus a comparatively smaller NN) as negative output nodes do not die.

NOTE: alpha = 1 will be a special case of parametric ReLu. There must be a balance between the amount of liveliness you want in the negative region vs the linearity of the activation function.