Let's say an image has 28*28 pixels, which leads to 784 input nodes in a feed-forward neural network. If an image can be classified into 1 of 10 numbers (e.g. [MNIST][1]), there are 10 output nodes.

We train (with gradient descent and back-propagation) the FFNN with a set of known pictures until we get a good accuracy.

Successively, we get a new training picture, which we want to use to train the FFNN even further. However, wouldn't this new training picture destroy the previously learned weights, which have been calibrated to recognize the former training pictures?

 [1]: https://en.wikipedia.org/wiki/MNIST_database