I have a large data set that doesn't fit in memory and would have to use something like Keras's model.fit_generator if I would like to train the model on all of the available data. The problem is that my data load time is larger than a single epoch and I would hate to incur that data load cost for each epoch.

The alternative approach that yields some value is to load as much data as possible, train the model for a few hundred epochs, then load the next portion of the data and reiterate for the same amount of epochs. And repeat this until all my data is "seen" by the model.

Intuitively I understand that this is sub-optimal as the model will tend to optimize for the latest portion of the data and "forget" the previous data but I would like a more in-depth explanation of the downsides of that method and if there are any ways to overcome them.

  • $\begingroup$ Data broken into batches might be better for generalization, just make sure all the batches are iterated over in a single epoch instead of going over the same batch multiple times and then discarding it permanently. $\endgroup$ – DuttaA May 11 '19 at 6:48
  • $\begingroup$ What do you mean by "data load time"? Do you mean the time required to load a batch of data or to load the whole dataset? $\endgroup$ – nbro May 11 '19 at 16:51
  • $\begingroup$ @DuttaA "Data broken into batches might be better for generalization", any evidence (paper) to support this claim? Batch-training mainly affects the training time (thus the convergence) and not much the performance (e.g. accuracy on the validation dataset) of the model. $\endgroup$ – nbro May 11 '19 at 16:55
  • $\begingroup$ @nbro see k-fold cross validation. $\endgroup$ – DuttaA May 11 '19 at 17:26
  • $\begingroup$ @DuttaA What does cross validation have to do with my question? CV is mainly used when you have small datasets and you want to exploit the whole data for training and validating. $\endgroup$ – nbro May 11 '19 at 17:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.