4
$\begingroup$

I've seen some comments in online articles/tutorials or Stack Overflow questions which suggest that increasing number of epochs can result in overfitting. But my intuition tells me that there should be no direct relationship at all between number of epochs and overfitting. So I'm looking for answer which explains if I'm right or wrong (or whatever's in between).

Here's my reasoning though. To overfit, you need to have enough free parameters (I think this is called "capacity" in neural networks) in your model to generate a function which can replicate the sample data points. If you don't have enough free parameters, you'll never overfit. You might just under fit.

So really, if you don't have too many free parameters, you could run infinite epochs and never overfit. If you have too many free parameters, then yes, the more epochs you have the more likely it is that you get to a place where you're overfitting. But that's just because running more epochs revealed the root cause: too many free parameters. The real loss function doesn't care about how many epochs you run. It existed the moment you defined your model structure, before you ever even tried to do gradient descent on it.

In fact, I'd venture as far as to say: assuming you have the computational resources and time, you should always aim to run as many epochs as possible, because that will tell you whether your model is prone to overfitting. Your best model will be the one that provides great training and validation accuracy, no matter how many epochs you run it for.

EDIT While reading more into this, I realise I forgot to take into account that you can arbitrarily vary the sample size as well. Given a fixed model, a smaller sample size is more prone to being overfit. And then that kind of makes me doubt my intuition above. Still happy to get an answer though!

$\endgroup$
4
  • 1
    $\begingroup$ Well you are definitely correct, but if you can select the right number of parameters for a given problem then probably you have solved the toughest problem in ML i.e. selecting of correct hyperparameters (one of which is the number of free parameters). $\endgroup$ – user9947 Dec 28 '19 at 15:50
  • $\begingroup$ @DuttaA haha well thanks. I'm about to post an edit which makes me feel not so definitely correct. Check it out in a minute $\endgroup$ – Alexander Soare Dec 28 '19 at 15:51
  • $\begingroup$ I think it also depends on how you train the network. If you have some kind of regulariser (i.e. you add noise to the loss) while training the net, you may not overfit. $\endgroup$ – nbro Dec 28 '19 at 16:27
  • $\begingroup$ Early stopping of training (before loss converge) can sometimes act as stochastic regularizer and decrease overfitting. That method is highly unreliable though and could cause "hyperparameter overfitting". That is relationships between number of epochs and overfitting. $\endgroup$ – mirror2image Dec 29 '19 at 6:39
2
$\begingroup$

The Problem of Overfitting

In most cases, when you increase a lot the number of epochs your model finally overfits. This is because your model reaches the point that it does not learn anymore but tries to remember what it has seen before. This is overfitting. So there is often a trade-off between the number of epochs and overfitting. In general a good way to avoid overfitting, except for fine-tuning, regularization, dropout etc, is to understand what you have from the learning curve. In most cases overfitting happens, after some epochs have passed, and, as a result the training error still decreases, whereas the validation error flactuates or increases. If so, you should save only the learning updates before overfitting appears and/or validation error is minimum.
Methods: early-stopping, checkpoint.

useful link: https://machinelearningmastery.com/learning-curves-for-diagnosing-machine-learning-model-performance/

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.