How can transfer learning be used to mitigate catastrophic forgetting. Could someone elaborate on this?
3
-
2$\begingroup$ I think this claim is false Specifically I do not see how it could make sense to define transfer learning, which is a feature, can be defined as a case for catastrophic forgetting which, as it also the name suggests, is an issue Could you please elaborate more on why this claim makes sense or do you have reference to share? $\endgroup$– Nicola BerniniAug 24, 2019 at 11:07
-
$\begingroup$ Well, I am not defining transfer learning. I am also not claiming anything. My thoughts' basis is: Transfer learning is aimed at using pre-trained network weights to apply to some new task by only updating the previously learned weights for a comparatively very small size of input data. On the contrary, catastrophic forgetting is the tendency of an artificial neural network to completely and abruptly forget previously learned information upon learning new information. I will update the question to make it sensible enough. $\endgroup$– naiveAug 24, 2019 at 11:44
-
$\begingroup$ See this answer. $\endgroup$– nbroOct 1, 2020 at 15:56
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
Transfer learning is a field where you apply knowledge from a source onto a target. This is a vague notion and there is an abundance of literature pertaining to it. Given your question I will work under the assumption that you are referring weight/architecture sharing between model (in other words training a model on one dataset and using it as a featurizer for another dataset)
Now any learning system without lossless memory will have remnants of catastrophic forgetting. So let's think about how we would implement this transfer and what effects can be derived from this.
- One implementation involves transferring a component and only training additional layers.
- Another is retraining the entire system but at a lower learning rate?
In setting 1, we can make the claim catastrophic forgetting is minimized by the fact that there is an unbiased featurizer that cant forget based on a sampling regime, though this does not mean additional layers which are still being trained can still faulter in this error mode.
In setting 2, we can make the claim catasrophic forgetting can be reduced compared to a normal end-to-end no-transfer training because the unbiased featurizers difference can be analytically bounded by its initial transferred featurization (complexity class is based on both the function and the number of steps -- so longer you train, the more likely it can forget)
These reasons are talking about mitigating and not erasing the concept of catastrophic forgetting, that is because as I mentioned above any learning system without lossless memory will have remnants of catastrophic forgetting, so making the generalized claim about transfer learning may not always fit the bill.
