Successively, we get a new training picture, which we want to use to train the FFNN even further. However, wouldn't this new training picture destroy the previously learned weights, which have been calibrated to recognize the former training pictures?
This can happen, and happen to various degrees depending on how the neural network is set up, but it is usually something you want to avoid.
Provided a neural network has enough capacity (in terms of number of free parameters), then the function it generates can be flexible enough to approximate the "true" function of the problem you are trying to match to data. Then as long as the learning steps are small enough, adjustments to learn one point on the function will take a larger step towards correct answer on that point than they adjust away from correct answer on other points (intuitively this is about the mutually good solutions being in part orthogonal and/or correlated, so a step towards A does not necessarily move the same distance away from B). Repeating the points again and again will allow the NN to match all the points over time.
Often this is possible to do almost exactly for a large amount of data at the same time. However, this is the opposite problem to your worries in the question, and is considered a bad thing because it can cause overfitting. Overfitting is very easy to achieve with neural networks because they are so flexible and can have large numbers of parameters.
Most often, the goal of training a supervised learner, such as a neural network, is not to exactly match the training data, but to generalise to new data from the same population distribution. You will typically spend far more time and effort combatting overfitting when training a NN, than worrying about lack of capacity to handle multiple data points. Methods to combat overfitting are:
- Regularisation. Techniques that adjust a learning algorithm, typically by adding some constraint.
- Cross validation. Measuring results of training against unseen data, and picking the best generalised model.
