Smallest possible network to approximate the $sin$ function

Question

The main goal is: Find the smallest possible neural network to approximate the $sin$ function.

Moreover, I want to find a qualitative reason why this network is the smallest possible network.

I have created 8000 random $x$ values with corresponding target values $sin(x)$. The network, which am currently considering, consists of 1 input neuron, 3 neurons in two hidden layers, and 1 output neuron:

Network architecture:

The neural network can be written as function $$y = sig(w_3 \cdot sig(w_1 \cdot x) + w_4 \cdot sig(w_2 \cdot x)),$$ where $\text{sig}$ is the sigmoid activation function.

$tanh$ activation function:
When I use $tanh$ as an activation function, the network is able to hit the 2 extrema of the $sin$ function:

$tanh$ activation function" />

Sigmoid activation function:
However, when I use the sigmoid activation function $\text{sig}$, only the first extremum is hit. The network output is not a periodic function but converges:

My questions are now:

• Why does one get a better approximation with the $tanh$ activation function? What is a qualitative argument for that?
• Why does one need at least 3 hidden neurons? What is the reason that the approximation with $tanh$ does not work anymore, if one uses only 2 hidden neurons?

I really appreciate all your ideas on this problem!

Answer 1

Before anything, the function you have wrote for the network lacks the bias variables (I'm sure you used bias to get those beautiful images, otherwise your tanh network had to start from zero).

Generally I would say it's impossible to have a good approximation of sinus with just 3 neurons, but if you want to consider one period of sinus, then you can do something. for clarity look at this picture:

I've write the code for this task in colab and you can find it here, and you can play with it if you want.

If you run the network several times you may get different results (because of different initializations) and you can see some of them at the Results section of the link above. What you showed us in the images above are just two possibilities. But it's interesting that you can get better results with tanh rather than sigmoid and if you want to know why, I highly recommend you to look at this lecture of CS231n. In summary it's because tanh has the negative part and the network can learn better with it.

But actually their power of approximation are almost similar because 2*sigmoid(1.5*x) - 1 almost looks the same as tanh(x) and you can find it by looking the picture below:

So why you can't get the same results as tanh? that's because tanh suits the problem better and if the network wants to get the same result as tanh with sigmoid it should learn their transformation parameters and learning these parameters makes the learning task harder. So It's not impossible to get the same result with sigmoid but it's harder. And to show you that its possible, I have set the parameters of the network using sigmoid manually and got the result below (you can get better results if you have more time):

At last if you want to know why you can't get the same result with 2 neurons instead of 3 neurons, it's better to understand what does the network do with 3 neurons.
If you look at the output of the first layer, you may see something like this (which are outputs of two neurons it has):

Then the next layer gets the difference between the output of these two neurons (which is like sinus) and applies sigmoid or tanh to it, and that's how you get a good result. But when you have just one neuron in the first layer, you can't imagine some scenario like this and approximating one period of sinus is out of it's ability (underfitting).