I am learning about incremental learning and read that rehearsal learning is retraining with old data. In essence, isn't this the exact same thing as batch learning (with stochastic gradient descent)? You train a model by passing in batches of data and redo this with a set number of epochs.
If I'm understanding rehearsal learning correctly, you do the exact same thing but with "new" data. Thus, the only difference is inconsistencies in the epoch number across data batches.
In rehearsal, you do not necessarily train with all old training data, but you can just use some of it [1], which you add to your current (or new) training data.
In batch learning, at every epoch, you typically train with all training data, every step with a different batch (or subset) of the training data; so, if you have $N$ training examples and your batch size is $M$, then you will have $\lfloor N/M \rfloor $ gradient descent steps.
There is also pseudo-rehearsal, which is an alternative to rehearsal that is useful when you may not have access to the previously used training data [1] (for example, because it is expensive to store).
