1
$\begingroup$

All the literature I read seems to indicate catastrophic forgetting affects only neural networks. Do other online/incremental algorithms not suffer from catastrophic forgetting (for example, SGDClassifier)? Why would that be the case?

$\endgroup$
0
$\begingroup$

To enlighten this, you need to understand the cause of the catastrophic forgetting. Fundamentally, the cause is an overlap in representations of different aspects of data in the learning model [Using Semi-Distributed Representations to Overcome Catastrophic Forgetting in Connectionlst Networks]. This can be explained easily in neural networks based on the representations of the data in hidden layers. However, in learning models such as SGD Classifier, we cannot see any explicit representation for input data, and we merely have a decision function to classify it. Hence, we cannot see any serious discussion about catastrophic forgetting in machine learning literature around other learning methods than neural networks.

By the way, this forgetting can happen in another form of the problem called "concept drift" [Understanding Concept Drifts]. If the data distribution will change during training a model (any learning model such as SGD Classifier), the model is likely to forget the first part of the data as it is fitted to the new distribution. The cause of such an issue is the capacity of the model for training different distributions of data. So, you can find many discussions in the literature on "Concept Drift" for all types of learning methods.

$\endgroup$
1
  • $\begingroup$ thank you for the reply. Just to make sure, is it that catastrophic forgetting can take place even when there is no concept drift? It would greatly help if you can illustrate an example at a very broad level for the case where catastrophic forgetting can take place but there is no concept drift. $\endgroup$ Aug 12 at 21:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.