How exactly can ReLUs approximate non-linear and curved functions?

Question

Currently, the most commonly used activation functions are ReLUs. So I answered this question What is the purpose of an activation function in neural networks? and, while writing the answer, it struck me, how exactly can ReLUs approximate a non-linear function?

By pure mathematical definition, sure, it's a non-linear function due to the sharp bend, but, if we confine ourselves to the positive or the negative portion of the $x$-axis only, then it's linear in those regions. Let's say we take the whole $x$-axis also, then also it's kinda linear (not in a strict mathematical sense), in the sense that it cannot satisfactorily approximate curved functions, like a sine wave ($0 \rightarrow 90$) with a single node hidden layer as is possible by a sigmoid activation function.

So, what is the intuition behind the fact that ReLUs are used in NNs, giving satisfactory performance? Are non-linear functions, like the sigmoid and the tanh, thrown in the middle of the NN sometimes?

I am not asking for the purpose of ReLUs, even though they are kind of linear.

---

As per @Eka's comment, the ReLu derives its capability from discontinuity acting in the deep layers of the NN. Does this mean that ReLUs are good, as long as we use it in deep NNs and not a shallow NN?

Answer 1

The outputs of a ReLU network are always "linear" and discontinuous. They can approximate curves, but it could take a lot of ReLU units. However, at the same time, their outputs will often be interpreted as a continuous, curved output.

Imagine you trained a neural network that takes $x^3$ and outputs $|x^3|$ (which is similar to a parabola). This is easy for the ReLU function to do perfectly. In this case, the output is curved.

But it is not actually curved. The inputs here are 'linearly' related to the outputs. All the neural network does is it takes the input and returns the absolute value of the input. It performs a 'linear', non-curved function. You can only see that the output is non-linear when you graph it against the original $x$-values (the $x$ in $x^3$).

So, when we plot the output on a graph and it looks curved, it's usually because we associated different x-values with the input, and then plotted the output as the $y$-coordinate in relation to those $x$-values.

Okay, so you want to know how you would smoothly model $\sin(x)$ using ReLU. The trick is that you don't want to put $x$ as the input. Instead put something curved in relation to $x$ as the input, like $x^3$. So, the input is $x^3$ and the output is $\sin(x)$. The reason why this would work is that it isn't computing the sine of the input - it's computing sine of the cube root of the input. It could never smoothly compute the sine of the input itself. To graph the output $\sin(x)$, put the original $x$ as the $x$ coordinate (don't put the input) and put the output as the $y$ coordinate.