6
$\begingroup$

The main goal is: Find the smallest possible neural network to approximate the $sin$ function.

Moreover, I want to find a qualitative reason why this network is the smallest possible network.

I have created 8000 random $x$ values with corresponding target values $sin(x)$. The network, which am currently considering, consists of 1 input neuron, 3 neurons in two hidden layers, and 1 output neuron:

Network architecture:

Network architecture

The neural network can be written as function $$y = sig(w_3 \cdot sig(w_1 \cdot x) + w_4 \cdot sig(w_2 \cdot x)),$$ where $\text{sig}$ is the sigmoid activation function.

$tanh$ activation function:
When I use $tanh$ as an activation function, the network is able to hit the 2 extrema of the $sin$ function:

*<span class=$tanh$ activation function" />

Sigmoid activation function:
However, when I use the sigmoid activation function $\text{sig}$, only the first extremum is hit. The network output is not a periodic function but converges:

Sigmoid activation function

My questions are now:

  • Why does one get a better approximation with the $tanh$ activation function? What is a qualitative argument for that?
  • Why does one need at least 3 hidden neurons? What is the reason that the approximation with $tanh$ does not work anymore, if one uses only 2 hidden neurons?

I really appreciate all your ideas on this problem!

$\endgroup$
  • 1
    $\begingroup$ Is the smallest network that can "approximate" the sin function not just $f(x) = \sigma(w\cdot x + b)$ where $w = 1$, $b=0$, and $\sigma(\cdot) = \sin(\cdot)$? $\endgroup$ – Cameron Chandler Oct 19 at 1:18
  • $\begingroup$ thank you for your comment. But I am looking for the smallest network with a tanh or sigmoid activation function $\endgroup$ – JavAlex Oct 19 at 19:11
  • $\begingroup$ Is x restricted to some interval ? $\endgroup$ – pasaba por aqui Oct 19 at 22:32
6
+100
$\begingroup$

Before anything, the function you have wrote for the network lacks the bias variables (I'm sure you used bias to get those beautiful images, otherwise your tanh network had to start from zero).

Generally I would say it's impossible to have a good approximation of sinus with just 3 neurons, but if you want to consider one period of sinus, then you can do something. for clarity look at this picture:

enter image description here

I've write the code for this task in colab and you can find it here, and you can play with it if you want.

If you run the network several times you may get different results (because of different initializations) and you can see some of them at the Results section of the link above. What you showed us in the images above are just two possibilities. But it's interesting that you can get better results with tanh rather than sigmoid and if you want to know why, I highly recommend you to look at this lecture of CS231n. In summary it's because tanh has the negative part and the network can learn better with it.

But actually their power of approximation are almost similar because 2*sigmoid(1.5*x) - 1 almost looks the same as tanh(x) and you can find it by looking the picture below: enter image description here

So why you can't get the same results as tanh? that's because tanh suits the problem better and if the network wants to get the same result as tanh with sigmoid it should learn their transformation parameters and learning these parameters makes the learning task harder. So It's not impossible to get the same result with sigmoid but it's harder. And to show you that its possible, I have set the parameters of the network using sigmoid manually and got the result below (you can get better results if you have more time): enter image description here

At last if you want to know why you can't get the same result with 2 neurons instead of 3 neurons, it's better to understand what does the network do with 3 neurons.
If you look at the output of the first layer, you may see something like this (which are outputs of two neurons it has):

enter image description here

Then the next layer gets the difference between the output of these two neurons (which is like sinus) and applies sigmoid or tanh to it, and that's how you get a good result. But when you have just one neuron in the first layer, you can't imagine some scenario like this and approximating one period of sinus is out of it's ability (underfitting).

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you very much for your detailed answer. One question: How did you come up with the manually set weights for the sigmoid approximation of the sinus? Did you figure them out just by trial and error/ educated guesses or did you use any system? $\endgroup$ – JavAlex Oct 23 at 9:06
  • $\begingroup$ Happy if it helped. It was combination of both. The ratio of the weights was trial, but I knew what the network should do and what the output should look like in each layer (as I described in the answer). So it's not hard to find the weights. Feel free to uncomment the related parts in the colab link and play with the weights and biases to see their affect on output of each layer. $\endgroup$ – amin Oct 23 at 9:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.