So, currently the most commonly used activation functions are Re-Lu's. So I answered this question What is the purpose of an activation function in Neural Networks? and while writing the answer it struck me, how exactly can Re-Lu's approximate non-linear function?

By pure mathematical definition, sure, its a non-linear function due to the sharp bend, but if we confine ourselves to the positive or the negative portion of the x-axis only, then its linear in those regions. Let's say we take the whole x-axis also, then also its kinda linear (not in strict mathematical sense) in the sense that it cannot satisfactorily approximate curvaceous functions like sine wave (0 --> 90) with a single node hidden layer as is possible by a sigmoid activation function.

So what is the intuition behind the fact that Re-Lu's are used in NN's, giving satisfactory performance (I am not asking the purpose of Re-lu's) even though they are kind of linear? Or are non linear functions like sigmoid and tanh thrown in the middle of the network sometimes?

EDIT: As per @Eka's comment Re-Lu derives its capability from discontinuity acting in the deep layers of Neural Net. Does this mean that Re-Lu's are good as long as we use it in Deep NN's and not a shallow NN?

  • 2
    $\begingroup$ I am no expert but found this link quora.com/… $\endgroup$ – Eka Mar 9 '18 at 11:20
  • 1
    $\begingroup$ @Eka nice link....but they are stating hard facts without giving a nice intuition $\endgroup$ – DuttaA Mar 9 '18 at 11:34
  • 3
    $\begingroup$ This is a guess; The relu's ability to approxoimate non-linear functions can be a result of its discontinuity property ie max(0,x) acting in deep layers of neural network. There is an openai research in which they computed non-linear functions using a deep linear networks here is the link blog.openai.com/nonlinear-computation-in-linear-networks $\endgroup$ – Eka Mar 9 '18 at 13:56

The outputs of a ReLU network are always "linear" and discontinuous. They can approximate curves, but it could take a lot of ReLU units. However, at the same time their outputs will often be interpreted as a continuous, curved output.

Imagine you trained a neural network that takes x3 and outputs |x3| (which is similar to a parabola). This is easy for the ReLU function to do perfectly. In this case the output is curved.

But it isn't actually curved. The inputs here are 'linearly' related to the outputs. All the neural network does is it takes the input and returns the absolute value of the input. It performs a 'linear', non-curved function. You can only see that the output is non-linear when you graph it against the original x-values (the x in x3).

So when we plot the output on a graph and it looks curved, it's usually because we associated different x-values with the input, and then plotted the output as the y-coordinate in relation to those x-values.

Okay, so you want to know how you would smoothly model sin(x) using ReLU. The trick is that you don't want to put x as the input. Instead put something curved in relation to x as the input, like x3. So the input is x3 and the output is sin(x). The reason why this would work is that it isn't computing sine of the input - it's computing sine of the cube root of the input. It could never smoothly compute the sine of the input itself. To graph the output sin(x), put the original x as the x coordinate (don't put the input) and put the output as the y coordinate.

| improve this answer | |
  • $\begingroup$ The approach you are giving would require many relus is my guess...You are discretizing the curve...So for finer curves we require more relus... Correct? $\endgroup$ – DuttaA Mar 11 '18 at 12:40
  • $\begingroup$ Not quite, IMO. If you take a bunch of different linear combinations of the input in the first layer then you will have many different types of curves available to piece together in the next layer, again with more linear combinations. This can make for a flawless output. Imagine for example if you cut and pasted the curves together only at their local extrema. Then the resulting curves would be almost perfectly continuous. It also depends exactly how fine you want the output to be. It's true though that there are better activation functions than ReLU due to their excessive linearity (e.g. ELU) $\endgroup$ – Default picture Mar 11 '18 at 18:50
  • $\begingroup$ I could not understand how we will we have many different curves available by a simple Re-lu..it will be just a combination of 2 straight lines $\endgroup$ – DuttaA Mar 12 '18 at 2:24
  • $\begingroup$ @DuttaA The straight lines are just the mapping function. You can map a curve to a curve using a Relu mapping. All you need is to have a curve as your input data. $\endgroup$ – Default picture Mar 12 '18 at 2:28
  • $\begingroup$ @DuttaA The input number line is a straight line and the output number line is a straight line, but that doesn't say anything about the shape of the data on the number line. The xy plane is rectangular euclidean space, but you can have curves inside of rectangular space. $\endgroup$ – Default picture Mar 12 '18 at 2:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.