0
$\begingroup$

TL;DR;

How to build a neural network that properly approximates the sine function with different ranges?

Context and Question:

From this question I decided to use the Sergey's answer, however I used a different range of values for $x$ which yielded a very poor estimation of the sine function. I noticed that for large values of the $x$ range it will provide incorrect results.

I plotted the r$^2$-score for different values of the $x$ range obtaining the following: R2 Scores

Am I missing someting? Why is this happening?, how can one build a neural network for approximating the sine function?

The Code:

Here is the code I used to create the plot in case you want to reproduce it

from sklearn.neural_network import MLPRegressor
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import r2_score
import numpy as np
import matplotlib.pyplot as plt

sineLengths = []
scores = []

for sineLength in np.arange(40, 60, 0.1):
    sineLengths.append(sineLength)
    X = []
    x = np.arange(0, sineLength, 0.1)
    y = np.sin(x)

    for i in range(len(x)):
        X.append([x[i]])

    X_train, X_test, y_train, y_test = train_test_split(X, y,random_state=1, test_size=0.2)
    sc_X = StandardScaler()
    X_trainscaled=sc_X.fit_transform(X_train)
    X_testscaled=sc_X.transform(X_test)

    reg = MLPRegressor(hidden_layer_sizes=tuple([64]*5), activation="relu" ,random_state=1, max_iter=20000).fit(X_trainscaled, y_train)
    y_pred=reg.predict(X_testscaled)
    r2score = r2_score(y_pred, y_test)
    print("The Score with ", r2score)
    scores.append(r2score)
$\endgroup$
5
  • 1
    $\begingroup$ If I am reading this correctly, your feature for X is five different x input values in a strict sequence. Why? It would mean you are not approximating $sin(x)$ but some other related function $f(x,y,z,a,b)$ but constraining the valid inputs to $f$ whilst expecting $f(\cdot,\cdot,\cdot,\cdot,b) = sin(b)$ $\endgroup$ Mar 2, 2022 at 18:20
  • $\begingroup$ @NeilSlater, Thanks for your comment. I'm working in predicting a signal value taking into account the immediate previous historic values, I figured out that I could expect $f(x,y,z,a,b) = f(x) + f(y) + f(z) + f(a) + f(b)$, then, in this specific example $f(x) = f(y) = f(z) = f(a)=0$ and $f(b) = sin(b)$. However I took your comment into account and updated the code in the post as well as the scores and the range. I also scaled the $X$ with fit_transform and transform, and obtained the samples with the same step across ranges. It still doesn't work. Am I forgetting something? $\endgroup$
    – Hans
    Mar 2, 2022 at 22:01
  • $\begingroup$ @NeilSlater, I also updated the plot, btw :) $\endgroup$
    – Hans
    Mar 2, 2022 at 22:03
  • $\begingroup$ In the previous code, you were not providing the 5 previous values of $y$ as a sequence to predict the next value of $y$, but the 4 previous values of $x$ plus the current value of $x$, which is very different from predicting the continuation of a sequence. Either way, thanks for making the edit and including code that matches the question better. $\endgroup$ Mar 2, 2022 at 22:18
  • $\begingroup$ @NeilSlater, haha, you're right, my bad. Thx for pointing that out, and I agree that now the edition reflects the question more clearly $\endgroup$
    – Hans
    Mar 2, 2022 at 22:38

1 Answer 1

1
$\begingroup$

Two things are happening here:

  • As you increase the number of $sin$ cycles that the neural network needs to approximate, the problem becomes harder for it to learn. The network - being 5 layers with 64 neurons each - actually has plenty of capacity to learn up to the approx 10 cycles you are trying to teach it. However, training it becomes progressively harder.

  • scikit-learn's MLPRegressor has a lot of hyperparameters, which are mostly set with sensible defaults, but it looks like* it is not set to try hard when training begins to stall and become more stochastic. The main issue with that is how quickly it will give up when cost function is not improving epoch-by-epoch, and it is common for measurable improvements to take multiple epochs, especially with a small data set (you only have a few hundred samples) when progress epoch-to-epoch can become quite noisy.

I managed to fix this and make training more robust by . . .

Increasing training set size slightly

  x = np.arange(0, sineLength, 0.05)

You could probably do a lot more here, the larger the set, the less noisy each improvement per epoch will be. But I wanted to use other hyperparameters to show what you might need to do if training is tougher and you cannot afford to easily increase the training set size.

Adding hyperparameters that made the fit routine try harder

    reg = MLPRegressor(hidden_layer_sizes=tuple([64]*5), activation="relu", 
      random_state=1, max_iter=2000, early_stopping=True, 
      validation_fraction=0.2, n_iter_no_change=200, 
      tol=1e-5).fit(X_trainscaled, y_train)

Notes:

  • I added using validation for early stopping, to avoid over-fitting. You should always use a validation set (in any non-toy exercise), although you could use other regularisation approaches. I did not check whether any of your bad scores were due to overfitting, but I don't think it was likely. This consumes more of the training data, which is another reason why I increased the size of the training set.
  • I increased n_iter_no_change significantly, to overcome the noise in the improvements per epoch. Small data sets may need many epochs in order to reveal the real trend in the training process.
  • I decreased tol, to allow for almost any improvement to be considered worthwhile.
  • I reduced max_iter to prevent training taking too long on this toy problem, and because I probably over-compensated with n_iter_no_change and tol changes. That meant I did get quite a few warnings of Maximum iterations (2000) reached and the optimization hasn't converged yet but the scores were nearly always very good after 2000 epochs, so I did not feel any need to increase that. Probably your original code was giving up after 100 or even less epochs because loss was not improving, and that was what gave you the bad r2 scores.

The above changes resulted in this graph for me: Graph of r2score against input range of function

. . . almost all r2score values were above 0.99, just one scored badly at 0.22, likely due to random chance. I could probably tune the hyperparameters further to make it perfect, or maybe double the size of the training set, but I will leave that as an exercise for you.


* This is the first time I have used MLPRegressor, hence "looks like". I am not certain what all the consequences are for how they have set defaults.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .