# Are neural networks prone to catastrophic forgetting?

Imagine you show a neural network a picture of a lion 100 times and label with "dangerous", so it learns that lions are dangerous.

Now imagine that previously you have shown it millions of images of lions and alternatively labeled it as "dangerous" and "not dangerous", such that the probability of a lion being dangerous is 50%.

But those last 100 times has pushed the neural network into being very positive about regarding the lion as "dangerous", thus ignoring the last million lessons.

Therefore, it seems there is a flaw in neural networks, in that they can change their mind too quickly based on recent evidence. Especially if that previous evidence was in the middle.

Is there a neural network model that keeps track of how much evidence it has seen? (Or would this be equivalent to letting the learning rate decrease by $$1/T$$ where $$T$$ is the number of trials?)

• I'm talking about supervised learning where the user tells the NN the lion is dangerous. – zooby Jul 10 '19 at 4:04
• This kinda happens to people too though. It's really scary how easy can one "unlearn" that something is dangerous after doing it several times without consequence, which is about equal as the scenario you have described with AI. – Tomáš Zato - Reinstate Monica Jul 11 '19 at 13:16
• Flagged as too broad. This is way too dependent on what recognition techniques are being used by the network. Obviously, sure, in some cases the network will "forget" but in other cases it wont. It should be extremely clear that any answers to this question should start and end with, "It depends". – 8protons Jul 11 '19 at 21:42
• To be fair, this is one of those "pick your poison" type deals. A NN that favors recent experiences over historical ones is prone to ignoring the past, but it's able to respond to recent developments. For example, suppose all lions suddenly turn mankiller overnight, then your NN that favors recent experiences will be much faster in picking up the new threat, as opposed to the slower NN which basicaly says "lions weren't always dangerous in the past, I conclude that nothing new is happening" until lions have been 100% dangerous for longer than you'd like (and many human deaths later) – Flater Jul 12 '19 at 9:40
• Also, AGI would have relevant error weighting - the two errors are not equally bad in outcome. – MSalters Jul 12 '19 at 15:11

Yes, indeed, neural networks are very prone to catastrophic forgetting (or interference). Currently, this problem is often ignored because neural networks are mainly trained offline (sometimes called batch training), where this problem does not often arise, and not online or incrementally, which is fundamental to the development of artificial general intelligence.

There are some people that work on continual lifelong learning in neural networks, which attempts to adapt neural networks to continual lifelong learning, which is the ability of a model to learn from a stream of data continually, so that they do not completely forget previously acquired knowledge while learning new information. See, for example, the paper Continual lifelong learning with neural networks: A review (2019), by German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, Stefan Wermter, which summarises the problems and existing solutions related to catastrophic forgetting of neural networks.

• Thanks! I will have a read of that paper you suggest. – zooby Jul 10 '19 at 15:24
• Was the infamous corruption of Microsoft's "Tay" chatbot an example of catastrophic forgetting? – user560822 Jul 10 '19 at 20:51
• @TKK I think this would be a good new question on the website! – nbro Jul 10 '19 at 20:54
• @TKK Are you going to ask it? If not, could somebody else do so? I really want to know the answer. – wizzwizz4 Jul 11 '19 at 7:39
• I'm pretty sure that the phrase "There are some people that work on continual lifelong learning in neural networks, which attempts to adapt neural networks to continual lifelong learning" was written by a neural network. – Moyli Jul 12 '19 at 18:22

Yes, the problem of forgetting older training examples is a characteristic of Neural Networks. I wouldn't call it a "flaw" though because it helps them be more adaptive and allows for interesting applications such as transfer learning (if a network remembered old training too well, fine tuning it to new data would be meaningless).

In practice what you want to do is to mix the training examples for dangerous and not dangerous so that it doesn't see one category in the beginning and one at the end.

A standard training procedure would work like this:

for e in epochs:
shuffle dataset
for x_batch, y_batch in dataset:
train neural_network on x_batxh, y_batch


Note that the shuffle at every epoch guarantees that the network won't see the same training examples in the same order every epoch and that the classes will be mixed

Now to answer your question, yes decreasing the learning rate would make the network less prone to forgetting its previous training, but how would this work in a non-online setting? In order for a network to converge it needs multiple epochs of training (i.e. seeing each sample in the dataset many times).

What you are describing sounds like it could be a deliberate case of fine-tuning.

There is a fundamental assumption that makes minibatch gradient descent work for learning problems: It is assumed that any batch or temporal window of consecutive batches forms a decent approximation of the true global gradient of the error function with respect to any parameterization of the model. If the error surface itself is moving in a big way, that would thwart the purposes of gradient descent--since gradient descent is a local refinement algorithm, all bets are off when you suddenly change the underlying distribution. In the example you cited, catastrophic forgetting seems like it would be an after-effect of having "forgotten" data points previously seen, and is either a symptom of the distribution having changed, or of under-representation in the data of some important phenomenon, such that it is rarely seen relative to its importance.

Experience replay from reinforcement learning is a relevant concept that transfers well to this domain. Here is a paper that explores this concept with respect to catastrophic forgetting. As long as sampling represents the true gradients sufficiently well (look at training sample balancing for this) and the model has enough parameters, the catastrophic forgetting problem is unlikely to occur. In randomly shuffled datasets with replacement, it is most likely to occur where datapoints of a particular class are so rare that they are unlikely to be included for a long time during training, effectively fine-tuning the model to a different problem until a matching sample is seen again.

To answer your question I'd say: Maybe in theory, but not in practice.

The problem is that you only consider a chronological/sequential training.

Only once have I used such sequential training method that is called online-training or Online Machine Learning. That was using the woppal wabbit library. It is a feature (not an issue like you consider) of this library to adapt chronologically to the input it is fed with.

I insist: in the case of that library Woppal Wabbit, it is a feature to adapt chronologically. It is wanted that when you start only telling him that lions are dangerous, then that it adapts consequently.

But in all any other cases from courses exercises, to kaggle competitions, I have used a randomized subset of my input data as training set. And this is really crucial:

It is an important part of Machine Learning which is called Cross Validation. It is the way to estimate how good the trained Neural Network really is.

So as to have a good estimation of the validity of your Neural Network you take a random subset of your training data, in short, you take something like 80% of you data for training, and with the remaining 20% you evaluate how often the trained Neural Network gives good predictions.

And one also cannot simply go away without Cross Validation, because of the need to detect Overfitting (which is another concern).

It may seem to you like a possible theorethical problem, but I tend to say that current cross validation methods usage make your concern irrelevant.