I'm training a neural network on some input data. I know that loss increasing may be related to:

  • overfitting, if the loss increases on test data (while still decreases on training data)
  • oscillations near the optimal point, if the learning rate is too big

However, I find that while for some input data the net makes good predictions, for other data the loss continues to increase, even if I only train on one data point and if the learning rate is fairly low. To me, it's quite strange that performing the training on only one point the loss continues to increase and not decreases; in fact, the only reason I can find for this is a big learning rate.

Can you think about some other reason?

  • 1
    $\begingroup$ by loss, do you mean validation/training error? or do you mean the actual loss function? its also hard to evaluate this without knowing the architecture involved.... $\endgroup$ – k.c. sayz 'k.c sayz' Aug 7 '20 at 2:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.