# Mathematical intuition for the use of Re-Lu's in Machine Learning

So, currently the most commonly used activation functions are Re-Lu's. So I answered this question What is the purpose of an activation function in Neural Networks? and while writing the answer it struck me, how exactly can Re-Lu's approximate non-linear function?

By pure mathematical definition, sure, its a non-linear function due to the sharp bend, but if we confine ourselves to the positive or the negative portion of the x-axis only, then its linear in those regions. Let's say we take the whole x-axis also, then also its kinda linear (not in strict mathematical sense) in the sense that it cannot satisfactorily approximate curvaceous functions like sine wave (0 --> 90) with a single node hidden layer as is possible by a sigmoid activation function.

So what is the intuition behind the fact that Re-Lu's are used in NN's, giving satisfactory performance (I am not asking the purpose of Re-lu's) even though they are kind of linear? Or are non linear functions like sigmoid and tanh thrown in the middle of the network sometimes?

EDIT: As per @Eka's comment Re-Lu derives its capability from discontinuity acting in the deep layers of Neural Net. Does this mean that Re-Lu's are good as long as we use it in Deep NN's and not a shallow NN?

• I am no expert but found this link quora.com/… – Eka Mar 9 '18 at 11:20
• @Eka nice link....but they are stating hard facts without giving a nice intuition – DuttaA Mar 9 '18 at 11:34
• This is a guess; The relu's ability to approxoimate non-linear functions can be a result of its discontinuity property ie max(0,x) acting in deep layers of neural network. There is an openai research in which they computed non-linear functions using a deep linear networks here is the link blog.openai.com/nonlinear-computation-in-linear-networks – Eka Mar 9 '18 at 13:56