So, currently the most commonly used activation functions are Re-Lu's. So I answered this question What is the purpose of an activation function in Neural Networks? and while writing the answer it struck me, how exactly can Re-Lu's approximate non-linear function?

By pure mathematical definition, sure, its a non-linear function due to the sharp bend, but if we confine ourselves to the positive or the negative portion of the x-axis only, then its linear in those regions. Let's say we take the whole x-axis also, then also its kinda linear (not in strict mathematical sense) in the sense that it cannot satisfactorily approximate curvaceous functions like sine wave (0 --> 90) with a single node hidden layer as is possible by a sigmoid activation function.

So what is the intuition behind the fact that Re-Lu's are used in NN's, giving satisfactory performance (I am not asking the purpose of Re-lu's) even though they are kind of linear? Or are non linear functions like sigmoid and tanh thrown in the middle of the network sometimes?

EDIT: As per @Eka's comment Re-Lu derives its capability from discontinuity acting in the deep layers of Neural Net. Does this mean that Re-Lu's are good as long as we use it in Deep NN's and not a shallow NN?

  • 2
    $\begingroup$ I am no expert but found this link quora.com/… $\endgroup$ – Eka Mar 9 '18 at 11:20
  • 1
    $\begingroup$ @Eka nice link....but they are stating hard facts without giving a nice intuition $\endgroup$ – DuttaA Mar 9 '18 at 11:34
  • 3
    $\begingroup$ This is a guess; The relu's ability to approxoimate non-linear functions can be a result of its discontinuity property ie max(0,x) acting in deep layers of neural network. There is an openai research in which they computed non-linear functions using a deep linear networks here is the link blog.openai.com/nonlinear-computation-in-linear-networks $\endgroup$ – Eka Mar 9 '18 at 13:56

The outputs of a ReLU network are always "linear" and discontinuous. They can approximate curves, but it could take a lot of ReLU units. However, at the same time their outputs will often be interpreted as a continuous, curved output.

Imagine you trained a neural network that takes x3 and outputs |x3| (which is similar to a parabola). This is easy for the ReLU function to do perfectly. In this case the output is curved.

But it isn't actually curved. The inputs here are 'linearly' related to the outputs. All the neural network does is it takes the input and returns the absolute value of the input. It performs a 'linear', non-curved function. You can only see that the output is non-linear when you graph it against the original x-values (the x in x3).

So when we plot the output on a graph and it looks curved, it's usually because we associated different x-values with the input, and then plotted the output as the y-coordinate in relation to those x-values.

Okay, so you want to know how you would smoothly model sin(x) using ReLU. The trick is that you don't want to put x as the input. Instead put something curved in relation to x as the input, like x3. So the input is x3 and the output is sin(x). The reason why this would work is that it isn't computing sine of the input - it's computing sine of the cube root of the input. It could never smoothly compute the sine of the input itself. To graph the output sin(x), put the original x as the x coordinate (don't put the input) and put the output as the y coordinate.

| improve this answer | |
  • $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$ – nbro Jun 25 at 11:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.