The ReLU activation function is defined as follows

$$y = \operatorname{max}(0,x)$$

And the linear activation function is defined as follows

$$y = x$$

The ReLU nonlinearity just clips the values less than 0 to 0 and passes everything else. Then why not to use a linear activation function instead, as it will pass all the gradient information during backpropagation? I do see that parametric ReLU (PReLU) does provide this possibility.

I just want to know if there is a proper explanation to using ReLU as default or it is just based on observations that it performs better on the training sets.

  • $\begingroup$ Use as interpretation of the activation function result the one of "error". Relu has a value 0, meaning expected optimal result is reached. Lineal has a minimum of less infinite, meaning the error can be less than zero (not applicable in most cases) and always posible to improve (near than never applicable) $\endgroup$ – pasaba por aqui May 19 '18 at 18:00
  • 1
    $\begingroup$ A simple intuition behind this, is that an ANN with all linear activations is analogous to linear regression $\endgroup$ – hisairnessag3 Feb 18 '19 at 10:30

The ReLu is a non-linear activation function. Check out this question for the intuition behind using ReLu's (also check out the comments). There is a very simple reason of why we do not use a linear activation function.

Say you have a feature vector $x_0$ and weight vector $W_1$. Passing through a layer in a Neural Net will give the output as

$W_1^T * x_0 = x_1$

(dot product of weights and input vector). Now passing the output through next layer will give you

$W_2^T * x_1 = x_2$

So expanding this we get

$x_2 = W_2^T * W_1^T * x_0 = W_2^T * W_1^T * x_0 = W_{compact}^T * x_0$

Thus as you can see there is a linear relationship between input and output, and the function we want to model is generally non-linear, and so we cannot model it.

You can check out my answer here on non-linear activation.

Parametric ReLu has few advantages over normal ReLu. Here is a great answer by @NeilSlater on the same. It is basically trying to tell us that if we use ReLu's we will end up with a lot of redundant or dead nodes in a Neural Net (those which have a negative output) which do not contribute to the result, and thus do not have a derivative. Thus to approximate a function we will require a larger NN, whereas parametric ReLu's absolve us of this problem,(thus a comparatively smaller NN) as negative output nodes do not die.

NOTE: alpha = 1 will be a special case of parametric ReLu. There must be a balance between the amount of liveliness you want in the negative region vs the linearity of the activation function.

| improve this answer | |

A multi-layer network in which all units have linear activation functions can always be collapsed to an equivalent network with two layers of units. That is why it is essential to use nonlinear unit activation functions.

The underlying reason for using nonlinear activation functions involves a remarkable theorem of Cybenko (1989), which states that one layer of nonlinear hidden units is sufficient to approximate any mapping from input to output units. Actually, I think there is a later proof which specifies that the nonlinearity can be any non-polynomial function (e.g. sigmoidal).

This text is based on the book: Artificial Intelligence Engines: A Tutorial Introduction to the Mathematics of Deep Learning.

| improve this answer | |
  • $\begingroup$ Links to the proof of Cybenko's theorem would be more useful to OP than links to your book, unless you can specify where in your book the topic is covered. Ideally your book outlines Cybenko's theorem? $\endgroup$ – Neil Slater Jan 24 at 13:19
  • $\begingroup$ The Cybenko's proof assumes that a sigmoid is used (if I remember correctly), but it's true that other proofs have been provided that are more general. $\endgroup$ – nbro Sep 24 at 10:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.