5
$\begingroup$

I've seen some comments in online articles/tutorials or Stack Overflow questions which suggest that increasing the number of epochs can result in overfitting. But my intuition tells me that there should be no direct relationship at all between the number of epochs and overfitting. So I'm looking for an answer which explains if I'm right or wrong (or whatever's in between).

Here's my reasoning though. To overfit, you need to have enough free parameters (I think this is called "capacity" in neural networks) in your model to generate a function that can replicate the sample data points. If you don't have enough free parameters, you'll never overfit. You might just underfit.

So really, if you don't have too many free parameters, you could run infinite epochs and never overfit. If you have too many free parameters, then yes, the more epochs you have the more likely it is that you get to a place where you're overfitting. But that's just because running more epochs revealed the root cause: too many free parameters. The real loss function doesn't care about how many epochs you run. It existed the moment you defined your model structure before you ever even tried to do gradient descent on it.

In fact, I'd venture as far as to say: assuming you have the computational resources and time, you should always aim to run as many epochs as possible because that will tell you whether your model is prone to overfitting. Your best model will be the one that provides great training and validation accuracy, no matter how many epochs you run it for.

EDIT While reading more into this, I realise I forgot to take into account that you can arbitrarily vary the sample size as well. Given a fixed model, a smaller sample size is more prone to overfitting. And then that kind of makes me doubt my intuition above. Still happy to get an answer though!

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ Well you are definitely correct, but if you can select the right number of parameters for a given problem then probably you have solved the toughest problem in ML i.e. selecting of correct hyperparameters (one of which is the number of free parameters). $\endgroup$
    – user9947
    Dec 28, 2019 at 15:50
  • $\begingroup$ @DuttaA haha well thanks. I'm about to post an edit which makes me feel not so definitely correct. Check it out in a minute $\endgroup$
    – Alexander Soare
    Dec 28, 2019 at 15:51
  • $\begingroup$ I think it also depends on how you train the network. If you have some kind of regulariser (i.e. you add noise to the loss) while training the net, you may not overfit. $\endgroup$
    – nbro
    Dec 28, 2019 at 16:27
  • $\begingroup$ Early stopping of training (before loss converge) can sometimes act as stochastic regularizer and decrease overfitting. That method is highly unreliable though and could cause "hyperparameter overfitting". That is relationships between number of epochs and overfitting. $\endgroup$
    – mirror2image
    Dec 29, 2019 at 6:39
  • $\begingroup$ If my answer was helpful, please check it as right answer. $\endgroup$
    – ddaedalus
    Sep 5, 2021 at 9:57

1 Answer 1

Reset to default
3
$\begingroup$

The Problem of Overfitting

In most cases, when you increase a lot the number of epochs your model finally overfits. This is because your model reaches the point that it does not learn anymore but tries to remember what it has seen before. This is overfitting. So there is often a trade-off between the number of epochs and overfitting. In general, a good way to avoid overfitting, except for fine-tuning, regularization, dropout, etc, is to understand what you have from the learning curve. In most cases, overfitting happens, after some epochs have passed, and, as a result, the training error still decreases, whereas the validation error fluctuates or increases. If so, you should save only the learning updates before overfitting appears and/or validation error is minimum.
Methods: early-stopping, checkpoint. useful link: https://machinelearningmastery.com/learning-curves-for-diagnosing-machine-learning-model-performance/

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .