Imagine you show a neural network a picture of a lion 100 times and label with "dangerous", so it learns that lions are dangerous.

Now imagine that previously you have shown it millions of images of lions and alternatively labeled it as "dangerous" and "not dangerous", such that the probability of a lion being dangerous is 50%.

But those last 100 times has pushed the neural network into being very positive about regarding the lion as "dangerous", thus ignoring the last million lessons.

Therefore, it seems there is a flaw in neural networks, in that they can change their mind too quickly based on recent evidence. Especially if that previous evidence was in the middle.

Is there a neural network model that keeps track of how much evidence it has seen? (Or would this be equivalent to letting the learning rate decrease by $1/T$ where $T$ is the number of trials?)

  • $\begingroup$ I'm talking about supervised learning where the user tells the NN the lion is dangerous. $\endgroup$ – zooby Jul 10 '19 at 4:04
  • $\begingroup$ This kinda happens to people too though. It's really scary how easy can one "unlearn" that something is dangerous after doing it several times without consequence, which is about equal as the scenario you have described with AI. $\endgroup$ – Tomáš Zato - Reinstate Monica Jul 11 '19 at 13:16
  • 2
    $\begingroup$ Flagged as too broad. This is way too dependent on what recognition techniques are being used by the network. Obviously, sure, in some cases the network will "forget" but in other cases it wont. It should be extremely clear that any answers to this question should start and end with, "It depends". $\endgroup$ – 8protons Jul 11 '19 at 21:42
  • 3
    $\begingroup$ To be fair, this is one of those "pick your poison" type deals. A NN that favors recent experiences over historical ones is prone to ignoring the past, but it's able to respond to recent developments. For example, suppose all lions suddenly turn mankiller overnight, then your NN that favors recent experiences will be much faster in picking up the new threat, as opposed to the slower NN which basicaly says "lions weren't always dangerous in the past, I conclude that nothing new is happening" until lions have been 100% dangerous for longer than you'd like (and many human deaths later) $\endgroup$ – Flater Jul 12 '19 at 9:40
  • 1
    $\begingroup$ Also, AGI would have relevant error weighting - the two errors are not equally bad in outcome. $\endgroup$ – MSalters Jul 12 '19 at 15:11

Yes, indeed, neural networks are very prone to catastrophic forgetting (or interference). Currently, this problem is often ignored because neural networks are mainly trained offline (sometimes called batch training), where this problem does not often arise, and not online or incrementally, which is fundamental to the development of artificial general intelligence.

There are some people that work on continual lifelong learning in neural networks, which attempts to adapt neural networks to continual lifelong learning, which is the ability of a model to learn from a stream of data continually, so that they do not completely forget previously acquired knowledge while learning new information. See, for example, the paper Continual lifelong learning with neural networks: A review (2019), by German I. Parisi, Ronald Kemker, Jose L. Part, Christopher Kanan, Stefan Wermter, which summarises the problems and existing solutions related to catastrophic forgetting of neural networks.

  • 1
    $\begingroup$ Thanks! I will have a read of that paper you suggest. $\endgroup$ – zooby Jul 10 '19 at 15:24
  • 7
    $\begingroup$ Was the infamous corruption of Microsoft's "Tay" chatbot an example of catastrophic forgetting? $\endgroup$ – user560822 Jul 10 '19 at 20:51
  • 4
    $\begingroup$ @TKK I think this would be a good new question on the website! $\endgroup$ – nbro Jul 10 '19 at 20:54
  • 2
    $\begingroup$ @TKK Are you going to ask it? If not, could somebody else do so? I really want to know the answer. $\endgroup$ – wizzwizz4 Jul 11 '19 at 7:39
  • 3
    $\begingroup$ I'm pretty sure that the phrase "There are some people that work on continual lifelong learning in neural networks, which attempts to adapt neural networks to continual lifelong learning" was written by a neural network. $\endgroup$ – Moyli Jul 12 '19 at 18:22

Yes, the problem of forgetting older training examples is a characteristic of Neural Networks. I wouldn't call it a "flaw" though because it helps them be more adaptive and allows for interesting applications such as transfer learning (if a network remembered old training too well, fine tuning it to new data would be meaningless).

In practice what you want to do is to mix the training examples for dangerous and not dangerous so that it doesn't see one category in the beginning and one at the end.

A standard training procedure would work like this:

for e in epochs:
    shuffle dataset
    for x_batch, y_batch in dataset:
        train neural_network on x_batxh, y_batch

Note that the shuffle at every epoch guarantees that the network won't see the same training examples in the same order every epoch and that the classes will be mixed

Now to answer your question, yes decreasing the learning rate would make the network less prone to forgetting its previous training, but how would this work in a non-online setting? In order for a network to converge it needs multiple epochs of training (i.e. seeing each sample in the dataset many times).


What you are describing sounds like it could be a deliberate case of fine-tuning.

There is a fundamental assumption that makes minibatch gradient descent work for learning problems: It is assumed that any batch or temporal window of consecutive batches forms a decent approximation of the true global gradient of the error function with respect to any parameterization of the model. If the error surface itself is moving in a big way, that would thwart the purposes of gradient descent--since gradient descent is a local refinement algorithm, all bets are off when you suddenly change the underlying distribution. In the example you cited, catastrophic forgetting seems like it would be an after-effect of having "forgotten" data points previously seen, and is either a symptom of the distribution having changed, or of under-representation in the data of some important phenomenon, such that it is rarely seen relative to its importance.

Experience replay from reinforcement learning is a relevant concept that transfers well to this domain. Here is a paper that explores this concept with respect to catastrophic forgetting. As long as sampling represents the true gradients sufficiently well (look at training sample balancing for this) and the model has enough parameters, the catastrophic forgetting problem is unlikely to occur. In randomly shuffled datasets with replacement, it is most likely to occur where datapoints of a particular class are so rare that they are unlikely to be included for a long time during training, effectively fine-tuning the model to a different problem until a matching sample is seen again.


Maybe in theory, but not in practice. The thing is you seem to consider only chronological/sequential training.

And there are two ways to view this issue:

  1. online learning -> then it is a feature of the method
  2. offline learning -> it does not happen thanks to several order randomizations

1. Online-Training or Online Machine Learning.

Using the woppal wabbit library. It is a feature (not an issue like you consider) of this library to adapt chronologically to the input it is fed with.

I insist: it is a feature to adapt chronologically. It is wanted that when you start only telling him that lions are dangerous, then that it adapts consequently.

2. Offline-Training

In my personal experience, I have used only randomized subsets of my input data as training set. And this randomization is crucial.

Randomizations happens namely:

  • during the training of the neural network, each epochs generally randomize the dataset order
  • during cross-validation, randomization is used as a way to evaluate a robust model that generalises well and does not overfit
  • $\begingroup$ This does not answer the OP's question ("Are neural networks prone to catastrophic forgetting?") at all. Except perhaps for the first line: "Maybe in theory, but not in practice." - which is incorrect. Neural networks are indeed prone to catastrophic forgetting. Please back up your claims with sources. $\endgroup$ – Mathias Müller Jan 29 at 8:28
  • $\begingroup$ @MathiasMüller I probably misunderstand your comment. Please could you explain how "Online Machine Learning" and "Cross Validation" are unrelated: 1/ to my statement 2/ to the original question . Thanks for detailing your contradiction just sufficiently. $\endgroup$ – Stephane Rolland Jan 29 at 9:04
  • $\begingroup$ Take notice that I talk about online training, while the accepted answer talks about "training offline" : "Currently, this problem is often ignored because neural networks are mainly trained offline (sometimes called batch training), where this problem does not often arise ." $\endgroup$ – Stephane Rolland Jan 29 at 9:08
  • $\begingroup$ My apologies if I there is a misunderstanding on my part, but cross validation is not a solution to catastrophic forgetting. Cross validation is a method for robust model selection. Regardless of whether cross-validation is used or not, catastrophic forgetting is not an issue in offline training settings because 1) the training set is fixed and 2) randomly shuffled so that for each epoch, examples are presented in a different order. $\endgroup$ – Mathias Müller Jan 29 at 9:52
  • 1
    $\begingroup$ "Do you have sources for this claim?" yes, of course. Here is a good answer: stats.stackexchange.com/a/272412/228948 $\endgroup$ – Mathias Müller Jan 30 at 17:36

Not the answer you're looking for? Browse other questions tagged or ask your own question.