I'm training a neural network on some input data. I know that loss increasing may be related to:
- overfitting, if the loss increases on test data (while still decreases on training data)
- oscillations near the optimal point, if the learning rate is too big
However, I find that while for some input data the net makes good predictions, for other data the loss continues to increase, even if I only train on one data point and if the learning rate is fairly low. To me, it's quite strange that performing the training on only one point the loss continues to increase and not decreases; in fact, the only reason I can find for this is a big learning rate.
Can you think about some other reason?