Are neural networks prone to catastrophic forgetting?

Question

Imagine you show a neural network a picture of a lion 100 times and label it with "dangerous", so it learns that lions are dangerous.

Now imagine that previously you have shown it millions of images of lions and alternatively labeled it as "dangerous" and "not dangerous", such that the probability of a lion being dangerous is 50%.

But those last 100 times have pushed the neural network into being very positive about regarding the lion as "dangerous", thus ignoring the last million lessons.

Therefore, it seems there is a flaw in neural networks, in that they can change their mind too quickly based on recent evidence. Especially if that previous evidence was in the middle.

Is there a neural network model that keeps track of how much evidence it has seen? (Or would this be equivalent to letting the learning rate decrease by $1/T$ where $T$ is the number of trials?)

Answer 1

Yes, indeed, neural networks are very prone to catastrophic forgetting (or interference). Currently, this problem is often ignored because neural networks are mainly trained offline (sometimes called batch training), where this problem does not often arise, and not online or incrementally, which is fundamental to the development of artificial general intelligence.

There are some people that work on continual lifelong learning in neural networks, which attempts to adapt neural networks to continual lifelong learning, which is the ability of a model to learn from a stream of data continually, so that they do not completely forget previously acquired knowledge while learning new information. See, for example, the paper Continual lifelong learning with neural networks: A review (2019), by German I. Parisi et al., which summarises the problems and existing solutions related to catastrophic forgetting of neural networks.

Answer 2

Yes, the problem of forgetting older training examples is a characteristic of Neural Networks. I wouldn't call it a "flaw" though because it helps them be more adaptive and allows for interesting applications such as transfer learning (if a network remembered old training too well, fine tuning it to new data would be meaningless).

In practice what you want to do is to mix the training examples for dangerous and not dangerous so that it doesn't see one category in the beginning and one at the end.

A standard training procedure would work like this:

for e in epochs:
    shuffle dataset
    for x_batch, y_batch in dataset:
        train neural_network on x_batxh, y_batch

Note that the shuffle at every epoch guarantees that the network won't see the same training examples in the same order every epoch and that the classes will be mixed

Now to answer your question, yes decreasing the learning rate would make the network less prone to forgetting its previous training, but how would this work in a non-online setting? In order for a network to converge it needs multiple epochs of training (i.e. seeing each sample in the dataset many times).

Answer 3

What you are describing sounds like it could be a deliberate case of fine-tuning.

There is a fundamental assumption that makes minibatch gradient descent work for learning problems: It is assumed that any batch or temporal window of consecutive batches forms a decent approximation of the true global gradient of the error function with respect to any parameterization of the model. If the error surface itself is moving in a big way, that would thwart the purposes of gradient descent--since gradient descent is a local refinement algorithm, all bets are off when you suddenly change the underlying distribution. In the example you cited, catastrophic forgetting seems like it would be an after-effect of having "forgotten" data points previously seen, and is either a symptom of the distribution having changed, or of under-representation in the data of some important phenomenon, such that it is rarely seen relative to its importance.

Experience replay from reinforcement learning is a relevant concept that transfers well to this domain. Here is a paper that explores this concept with respect to catastrophic forgetting. As long as sampling represents the true gradients sufficiently well (look at training sample balancing for this) and the model has enough parameters, the catastrophic forgetting problem is unlikely to occur. In randomly shuffled datasets with replacement, it is most likely to occur where datapoints of a particular class are so rare that they are unlikely to be included for a long time during training, effectively fine-tuning the model to a different problem until a matching sample is seen again.

Answer 4

Maybe in theory, but not in practice. The thing is you seem to consider only chronological/sequential training.

And there are two ways to view this issue:

1. online learning -> then it is a feature of the method
2. offline learning -> it does not happen thanks to several order randomizations

---

1. Online-Training or Online Machine Learning.

Using the woppal wabbit library. It is a feature (not an issue like you consider) of this library to adapt chronologically to the input it is fed with.

I insist: it is a feature to adapt chronologically. It is wanted that when you start only telling him that lions are dangerous, then that it adapts consequently.

---

2. Offline-Training

In my personal experience, I have used only randomized subsets of my input data as training set. And this randomization is crucial.

Randomizations happens namely:

• during the training of the neural network, each epochs generally randomize the dataset order
• during cross-validation, randomization is used as a way to evaluate a robust model that generalises well and does not overfit