4
$\begingroup$

ReLU : y = max(0,x)

Linear : y = x

The ReLU nonlinearity just clips the values < 0 to 0 and passes everything else; then why not to use a linear activation function instead as it will pass all the gradient information during backpropagation.

I do see that Parametric ReLU (PReLU) does provide this possiblity.

I just want to know if there is a proper explanation to using ReLU as default or it is just based on observations that it performs better on the training sets.

$\endgroup$
  • $\begingroup$ Use as interpretation of the activation function result the one of "error". Relu has a value 0, meaning expected optimal result is reached. Lineal has a minimum of less infinite, meaning the error can be less than zero (not applicable in most cases) and always posible to improve (near than never applicable) $\endgroup$ – pasaba por aqui May 19 '18 at 18:00
  • $\begingroup$ This question should be migrated to: datascience.stackexchange.com. $\endgroup$ – JahKnows Jun 1 '18 at 8:47
  • 1
    $\begingroup$ A simple intuition behind this, is that an ANN with all linear activations is analogous to linear regression $\endgroup$ – hisairnessag3 Feb 18 '19 at 10:30
2
$\begingroup$

The ReLu is a non-linear activation function. Check out this question for the intuition behind using ReLu's (also check out the comments). There is a very simple reason of why we do not use a linear activation function.

Say you have a feature vector $x_0$ and weight vector $W_1$. Passing through a layer in a Neural Net will give the output as

$W_1^T * x_0 = x_1$

(dot product of weights and input vector). Now passing the output through next layer will give you

$W_2^T * x_1 = x_2$

So expanding this we get

$x_2 = W_2^T * W_1^T * x_0 = W_2^T * W_1^T * x_0 = W_{compact}^T * x_0$

Thus as you can see there is a linear relationship between input and output, and the function we want to model is generally non-linear, and so we cannot model it.

You can check out my answer here on non-linear activation.

Parametric ReLu has few advantages over normal ReLu. Here is a great answer by @NeilSlater on the same. It is basically trying to tell us that if we use ReLu's we will end up with a lot of redundant or dead nodes in a Neural Net (those which have a negative output) which do not contribute to the result, and thus do not have a derivative. Thus to approximate a function we will require a larger NN, whereas parametric ReLu's absolve us of this problem,(thus a comparatively smaller NN) as negative output nodes do not die.

NOTE: alpha = 1 will be a special case of parametric ReLu. There must be a balance between the amount of liveliness you want in the negative region vs the linearity of the activation function.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.