Two things are happening here:
As you increase the number of $sin$ cycles that the neural network needs to approximate, the problem becomes harder for it to learn. The network - being 5 layers with 64 neurons each - actually has plenty of capacity to learn up to the approx 10 cycles you are trying to teach it. However, training it becomes progressively harder.
MLPRegressor has a lot of hyperparameters, which are mostly set with sensible defaults, but it looks like* it is not set to try hard when training begins to stall and become more stochastic. The main issue with that is how quickly it will give up when cost function is not improving epoch-by-epoch, and it is common for measurable improvements to take multiple epochs, especially with a small data set (you only have a few hundred samples) when progress epoch-to-epoch can become quite noisy.
I managed to fix this and make training more robust by . . .
Increasing training set size slightly
x = np.arange(0, sineLength, 0.05)
You could probably do a lot more here, the larger the set, the less noisy each improvement per epoch will be. But I wanted to use other hyperparameters to show what you might need to do if training is tougher and you cannot afford to easily increase the training set size.
Adding hyperparameters that made the fit routine try harder
reg = MLPRegressor(hidden_layer_sizes=tuple(*5), activation="relu",
random_state=1, max_iter=2000, early_stopping=True,
- I added using validation for early stopping, to avoid over-fitting. You should always use a validation set (in any non-toy exercise), although you could use other regularisation approaches. I did not check whether any of your bad scores were due to overfitting, but I don't think it was likely. This consumes more of the training data, which is another reason why I increased the size of the training set.
- I increased
n_iter_no_change significantly, to overcome the noise in the improvements per epoch. Small data sets may need many epochs in order to reveal the real trend in the training process.
- I decreased
tol, to allow for almost any improvement to be considered worthwhile.
- I reduced
max_iter to prevent training taking too long on this toy problem, and because I probably over-compensated with
tol changes. That meant I did get quite a few warnings of
Maximum iterations (2000) reached and the optimization hasn't converged yet but the scores were nearly always very good after 2000 epochs, so I did not feel any need to increase that. Probably your original code was giving up after 100 or even less epochs because loss was not improving, and that was what gave you the bad r2 scores.
The above changes resulted in this graph for me:
. . . almost all r2score values were above 0.99, just one scored badly at 0.22, likely due to random chance. I could probably tune the hyperparameters further to make it perfect, or maybe double the size of the training set, but I will leave that as an exercise for you.
* This is the first time I have used
MLPRegressor, hence "looks like". I am not certain what all the consequences are for how they have set defaults.