I have a large data set that doesn't fit in memory and would have to use something like Keras's
model.fit_generator if I would like to train the model on all of the available data. The problem is that my data load time is larger than a single epoch and I would hate to incur that data load cost for each epoch.
The alternative approach that yields some value is to load as much data as possible, train the model for a few hundred epochs, then load the next portion of the data and reiterate for the same amount of epochs. And repeat this until all my data is "seen" by the model.
Intuitively I understand that this is sub-optimal as the model will tend to optimize for the latest portion of the data and "forget" the previous data but I would like a more in-depth explanation of the downsides of that method and if there are any ways to overcome them.