How can transfer learning be used to mitigate catastrophic forgetting. Could someone elaborate on this?
Transfer learning is a field where you apply knowledge from a source onto a target. This is a vague notion and there is an abundance of literature pertaining to it. Given your question I will work under the assumption that you are referring weight/architecture sharing between model (in other words training a model on one dataset and using it as a featurizer for another dataset)
Now any learning system without lossless memory will have remnants of catastrophic forgetting. So let's think about how we would implement this transfer and what effects can be derived from this.
- One implementation involves transferring a component and only training additional layers.
- Another is retraining the entire system but at a lower learning rate?
In setting 1, we can make the claim catastrophic forgetting is minimized by the fact that there is an unbiased featurizer that cant forget based on a sampling regime, though this does not mean additional layers which are still being trained can still faulter in this error mode.
In setting 2, we can make the claim catasrophic forgetting can be reduced compared to a normal end-to-end no-transfer training because the unbiased featurizers difference can be analytically bounded by its initial transferred featurization (complexity class is based on both the function and the number of steps -- so longer you train, the more likely it can forget)
These reasons are talking about mitigating and not erasing the concept of catastrophic forgetting, that is because as I mentioned above any learning system without lossless memory will have remnants of catastrophic forgetting, so making the generalized claim about transfer learning may not always fit the bill.