Should the training data be the same in each epoch?

If the training data is generated on the fly, for example, is there a difference between training 1000 samples with 1 epoch or training 1000 epochs with 1 sample each?

To elaborate further, samples do not need to be saved or stay in memory if they are never used again. However, if training performs best by training over the same samples repeatedly, then data would have to be stored to be reused in each epoch.

More samples is generally considered advantageous. Is there a disadvantage to never seeing the same sample twice in training?

  • 1
    $\begingroup$ Can you please clarify what you mean by "data is generated on the fly"? Do you mean that at each epoch new data is sampled from the data-generating process? $\endgroup$ – nbro Jul 14 at 0:27
  • $\begingroup$ An epoch refers to running over the entire training set. So for an epoch to actually be an epoch, the data must be the same. If the data changes each epoch, you aren't running epochs, but rather iterations. I'm confused as to why there are answers suggesting otherwise. $\endgroup$ – Recessive Jul 14 at 5:37
  • $\begingroup$ @nbro that is correct. Every sample could be unique for a virtually unlimited data stream. But if training requires that a specific set of samples must be seen repeatedly, then that data must be saved to be used again. $\endgroup$ – NewEndian Jul 14 at 16:28

Let's quickly get out our copies of Deep Learning by Goodfellow et al. (2016). More specifically, I'm referring to page 276.

On this page, the authors argue for a relatively small minibatch size, since there are less than linear returns for estimating the gradient when increasing the minibatch size. Returns here refer to the reduction of the standard error of the mean (gradient per weight) computed over a minibatch.

So, yes. In theory, having unlimited resources, you will get the best performance when averaging the loss over all samples in your dataset. In practice, however, the larger the size of minibatches, the slower the training procedure, and consequently the less the total number of weight updates that can be afforded. Reversely, in practice, the cheaper the weight updates, the quicker the training procedure can converge to a (subjectively) satisfactory result.

Eventually, also Goodfellow et al. state that rapidly computing gradients leads to much faster convergence (in terms of total computations) for most optimization algorithms than when training them more slowly on exact gradients.

So, to summarize: If the main concern is to get to a specific level of accuracy at all, go for rather low minibatch sizes, whereas you could go up to a few hundreds (as the Goodfellow et al. state as a reasonable upper bound on page 148) if you are interested in more accurate gradients for your weight updates.

| improve this answer | |
  • $\begingroup$ Thank you for the reference to this book. I will read it $\endgroup$ – NewEndian Jul 14 at 16:24

This would be more suitable as a comment but I don't have enough points; but here's my opinion.

Optimisation algorithms like gradient descent are iterative algorithms. So it is rarely possible that they arrive at the minima in 1 epoch. A single epoch means that all data points have been visited once or a certain number of data samples have been taken from a distribution. However more passes might be necessary.

generated on the fly

I am assuming that the data is being generated as a part of a fixed distribution. Hence multiple epochs of multiple samples is still the ideal scenario.

1000 samples 1 epoch: Not enough training.
1 sample 1000 epoch: Overfitting or possibly not enough training.

| improve this answer | |
  • $\begingroup$ How does this answer the question? $\endgroup$ – nbro Jul 13 at 23:32
  • $\begingroup$ My apologies @nbro. I think now the answer is more suitable. $\endgroup$ – kgkmeekg Jul 13 at 23:37
  • $\begingroup$ I think you need to explain a bit more.... Answer seems off, although, seems correct to me. $\endgroup$ – abhas_RewCie Jul 16 at 16:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.