Trying to understand why nonlinearity is important for neural networks by analogy

Question

Is the reason why linear activation functions are usually pretty bad at approximating functions the same reason why combinations of hermitian polynomials or combinations of sines and cosines are better at approximating a function than combinations of linear functions?

For example, regardless of the amount of terms in this combination of linear functions, the function will always be some form of $y = mx + b$. However, if we're summing sines, you absolutely cannot express a combination of sines and cosines as something of the form $A \sin{bx}$. For example, a combination of three sinusoids cannot be simplified further than $A \sin{bx} + B \sin{cx} + D \sin{ex}$.

Is this fact essentially why the Fourier series is able to approximate functions (other than obviously the fact that $A \sin{bx}$ is orthogonal to $B \sin{cx}$)? Because if it could be simplified into one sinusoid, it could never approximate an arbitrary function because it's lost its robustness? Because with other terms combined, whereas linear functions summed up gain no further ability to approximate, things like sinusoids actually begin to approximate really well with enough terms and with the right constants.

In that vein, is this the reason why non-linear activiation functions (also called non-linear classifiers?) are generally valued more than linear ones? Because linear activation functions simply are lousy function approximators, while, with enough constants and terms, non-linear activation functions can approximate any function?

Answer 1

Your analogy is correct, except it is not really an "analogy". Sin is an activation function - in past works (before modern deep learning boom) it was rather standard to see it listed as a possible activator.

So your expression $\sigma(x) = A\sin ax + B \sin bx + D \sin ex$ is of a neural network with one 3-neuron layer and a single output linear neuron:

$$\sigma(x) = \sum_i V_i \sin\left(\sum_jW_{ij} x_j + b_i\right)+\beta$$

With all biases being zero $b_i=\beta=0$, output weights are $V_i = \left(A,B,C\right)$ and the inner weight matrix is diagonal: $$W_{ij} = \begin{pmatrix}a&0&0\\0&b&0\\0&0&e\end{pmatrix}$$

Answer 2

Ok. Here is an analogy for you. The equation for a neuron is wx + b, which is equivalent to a straight line. If we don't apply non-linearity we will be stuck with a straight line forever. So, this type of network won't be even able to model points in a unit circle randomly distributed.

What does non-linearity do? If you look the graphs for x to the power 2, 3, 4 and so on. You see with each increase in power, the line gets tugged like a sine or cosine curve. That bend in the straight line allows us to then model boundaries with arbitrary shapes.

The more the difficult the boundary to model between the classes the more bends in the line you need to model and so, you keep on increasing the layers in neural network.