1
$\begingroup$

The problem to solve is non-linear regression of a non-linear function. My actual problem is to model the function "find the max over many quadratic forms": max(w.H.TQw), but to get started and to learn more about neural networks, I created a toy example for a non-linear regression task , using Pytorch. The problem is that the network never learns the function in a satisfactory way, even though my model is quite large with multiple layers (see below). Or is it not large enough or too large? How can the network be improved or may be even simplified to get a much smaller training error?

I experimented with different network architectures, but the result is never satisfactory. Usually the error is quite small within the input interval around around 0, but the network is not able to get good weights for the regions at the boundary of the interval (see plots below). The loss does not improve after a certain number of epochs. I could generate even more training data, but I have not yet understood completely, how the training can be improved (tuning parameters such as batch size, amount of data, number of layers, normalizing input (output?) data, number of neurons, epochs etc.)

My neural network has 8 layers with the following number of neurons: 1, 80,70,60, 40,40,20, 1

For the moment I do not care too much about overfitting, my goal is to understand, why a certain network architecture/certain hyper parameters need to be chosen. Of course, avoiding overfitting at the same time would be a bonus.

I am especially interested in using neural networks for regression tasks or as function approximators. In principle my problem should be able to be approximated to arbitrary accuracy by a single layer neural network, according to the universal approximation theorem, isn’t this correct?

Difference between trained model and original data

Loss value vs iterations

Difference between trained model and original data

$\endgroup$
  • 1
    $\begingroup$ Welcome to ai.se..instead of presenting the code you can just add the parameters of your NN since this site doesn't encourage questions about implementation details...also it is better to give a brief idea about the info conveyed by the graphs $\endgroup$ – DuttaA Apr 9 '18 at 11:44
  • $\begingroup$ Thank you, can you recommend a place to ask for implementation details? $\endgroup$ – BPresent Apr 9 '18 at 18:39
  • $\begingroup$ You can try crossvalidated or data science $\endgroup$ – DuttaA Apr 10 '18 at 1:50
0
$\begingroup$

Neural networks learn badly with large input ranges. Scale your inputs to a smaller range e.g. -2 to 2, and convert to/from this range to represent your function interval consistently.

$\endgroup$
  • $\begingroup$ My reputation is not high enough to upvote you, but the normalization helped! Thanks $\endgroup$ – BPresent Apr 9 '18 at 18:35
  • $\begingroup$ Are you sure just linearly putting values in another range conduces better results? It seems that there is nothing qualitatively different from the two. $\endgroup$ – Daniel Sep 14 '18 at 22:57
  • $\begingroup$ @Daniel: Yes I am sure. It can make a huge difference in practice. Always check the range of your inputs to a NN and scale them if necessary. The numerically ideal scale is around mean 0, standard deviation 1. There is some flexibility around that, but if you have inputs with values an order of magnitude or more different to that, the NN learning will suffer. $\endgroup$ – Neil Slater Sep 14 '18 at 23:39
  • $\begingroup$ @NeilSlater Oh, I thought you meant you meant just scaling the values between an interval and not normalizing them. $\endgroup$ – Daniel Sep 15 '18 at 21:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.