Do other online/incremental algorithms not suffer from catastrophic forgetting?

Question

All the literature I read seems to indicate catastrophic forgetting affects only neural networks. Do other online/incremental algorithms not suffer from catastrophic forgetting (for example, SGDClassifier)? Why would that be the case?

Answer 1

To enlighten this, you need to understand the cause of the catastrophic forgetting. Fundamentally, the cause is an overlap in representations of different aspects of data in the learning model [Using Semi-Distributed Representations to Overcome Catastrophic Forgetting in Connectionlst Networks]. This can be explained easily in neural networks based on the representations of the data in hidden layers. However, in learning models such as SGD Classifier, we cannot see any explicit representation for input data, and we merely have a decision function to classify it. Hence, we cannot see any serious discussion about catastrophic forgetting in machine learning literature around other learning methods than neural networks.

By the way, this forgetting can happen in another form of the problem called "concept drift" [Understanding Concept Drifts]. If the data distribution will change during training a model (any learning model such as SGD Classifier), the model is likely to forget the first part of the data as it is fitted to the new distribution. The cause of such an issue is the capacity of the model for training different distributions of data. So, you can find many discussions in the literature on "Concept Drift" for all types of learning methods.