3
$\begingroup$

I have a question about the training sequence regarding Neural Network recognition. Let's say an image has 28*28 pixels, which leads to 784 Input Nodes with various greyscale values and 10 output nodes, if the image shows a number 0-9. Then when the training data is used for a set of known pictures of numbers, the weights and hidden layers uses forward- and backpropagation to get get the proper hidden and weight layers for the known layer. However, doesn't a new training picture destroy the trained and balanced weights and nodes values? Because the weights and hidden nodes have been calibrated to recognize the former training picture? Thank you for assistance.

Kind regards David

$\endgroup$
  • $\begingroup$ I kindly request you to go through the community guidelines and Tour $\endgroup$ – quintumnia Apr 29 '18 at 8:34
1
$\begingroup$

However, doesn't a new training picture destroy the trained and balanced weights and nodes values? Because the weights and hidden nodes have been calibrated to recognize the former training picture?

This can happen, and happen to various degrees depending on how the neural network is set up, but it is usually something you want to avoid.

Provided a neural network has enough capacity (in terms of number of free parameters), then the function it generates can be flexible enough to approximate the "true" function of the problem you are trying to match to data. Then as long as the learning steps are small enough, adjustments to learn one point on the function will take a larger step towards correct answer on that point than they adjust away from correct answer on other points (intuitively this is about the mutually good solutions being in part orthogonal and/or correlated, so a step towards A does not necessarily move the same distance away from B). Repeating the points again and again will allow the NN to match all the points over time.

Often this is possible to do almost exactly for a large amount of data at the same time. However, this is the opposite problem to your worries in the question, and is considered a bad thing because it can cause overfitting. Overfitting is very easy to achieve with neural networks because they are so flexible and can have large numbers of parameters.

Most often, the goal of training a supervised learner, such as a neural network, is not to exactly match the training data, but to generalise to new data from the same population distribution. You will typically spend far more time and effort combatting overfitting when training a NN, than worrying about lack of capacity to handle multiple data points. Methods to combat overfitting are:

  • Regularisation. Techniques that adjust a learning algorithm, typically by adding some constraint.
  • Cross validation. Measuring results of training against unseen data, and picking the best generalised model.
$\endgroup$
  • $\begingroup$ Regularization isn't particularly effective for NN's, better add dropout layers. $\endgroup$ – DuttaA Aug 27 '18 at 19:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.