13
$\begingroup$

Currently, the most commonly used activation functions are ReLUs. So I answered this question What is the purpose of an activation function in neural networks? and, while writing the answer, it struck me, how exactly can ReLUs approximate a non-linear function?

By pure mathematical definition, sure, it's a non-linear function due to the sharp bend, but, if we confine ourselves to the positive or the negative portion of the $x$-axis only, then it's linear in those regions. Let's say we take the whole $x$-axis also, then also it's kinda linear (not in a strict mathematical sense), in the sense that it cannot satisfactorily approximate curved functions, like a sine wave ($0 \rightarrow 90$) with a single node hidden layer as is possible by a sigmoid activation function.

So, what is the intuition behind the fact that ReLUs are used in NNs, giving satisfactory performance? Are non-linear functions, like the sigmoid and the tanh, thrown in the middle of the NN sometimes?

I am not asking for the purpose of ReLUs, even though they are kind of linear.


As per @Eka's comment, the ReLu derives its capability from discontinuity acting in the deep layers of the NN. Does this mean that ReLUs are good, as long as we use it in deep NNs and not a shallow NN?

$\endgroup$
1
  • 3
    $\begingroup$ This is a guess; The relu's ability to approxoimate non-linear functions can be a result of its discontinuity property ie max(0,x) acting in deep layers of neural network. There is an openai research in which they computed non-linear functions using a deep linear networks here is the link blog.openai.com/nonlinear-computation-in-linear-networks $\endgroup$
    – Eka
    Mar 9, 2018 at 13:56

1 Answer 1

5
$\begingroup$

The outputs of a ReLU network are always "linear" and discontinuous. They can approximate curves, but it could take a lot of ReLU units. However, at the same time, their outputs will often be interpreted as a continuous, curved output.

Imagine you trained a neural network that takes $x^3$ and outputs $|x^3|$ (which is similar to a parabola). This is easy for the ReLU function to do perfectly. In this case, the output is curved.

But it is not actually curved. The inputs here are 'linearly' related to the outputs. All the neural network does is it takes the input and returns the absolute value of the input. It performs a 'linear', non-curved function. You can only see that the output is non-linear when you graph it against the original $x$-values (the $x$ in $x^3$).

So, when we plot the output on a graph and it looks curved, it's usually because we associated different x-values with the input, and then plotted the output as the $y$-coordinate in relation to those $x$-values.

Okay, so you want to know how you would smoothly model $\sin(x)$ using ReLU. The trick is that you don't want to put $x$ as the input. Instead put something curved in relation to $x$ as the input, like $x^3$. So, the input is $x^3$ and the output is $\sin(x)$. The reason why this would work is that it isn't computing the sine of the input - it's computing sine of the cube root of the input. It could never smoothly compute the sine of the input itself. To graph the output $\sin(x)$, put the original $x$ as the $x$ coordinate (don't put the input) and put the output as the $y$ coordinate.

$\endgroup$
2
  • $\begingroup$ Comments are not for extended discussion; this conversation has been moved to chat. $\endgroup$
    – nbro
    Jun 25, 2020 at 11:58
  • $\begingroup$ This is equivalent to adding an additional layer after the input which computes x^3 from x. This is okay when you know what you need to compute but how do you decide this in general? $\endgroup$ Aug 26, 2022 at 9:53

You must log in to answer this question.