So, currently the most commonly used activation functions are Re-Lu's. So I answered this question What is the purpose of an activation function in Neural Networks? and while writing the answer it struck me, how exactly can Re-Lu's approximate non-linear function?
By pure mathematical definition, sure, its a non-linear function due to the sharp bend, but if we confine ourselves to the positive or the negative portion of the x-axis only, then its linear in those regions. Let's say we take the whole x-axis also, then also its kinda linear (not in strict mathematical sense) in the sense that it cannot satisfactorily approximate curvaceous functions like sine wave (
0 --> 90) with a single node hidden layer as is possible by a sigmoid activation function.
So what is the intuition behind the fact that Re-Lu's are used in NN's, giving satisfactory performance (I am not asking the purpose of Re-lu's) even though they are kind of linear? Or are non linear functions like sigmoid and tanh thrown in the middle of the network sometimes?
EDIT: As per @Eka's comment Re-Lu derives its capability from discontinuity acting in the deep layers of Neural Net. Does this mean that Re-Lu's are good as long as we use it in Deep NN's and not a shallow NN?