1
$\begingroup$

I am a machine learning newbie. I am trying to understand backpropagation algorithm. I have a training dataset of 60 instances/records. So what is the correct order of the process:

  • Forward pass of the first instance. calculate the error.

  • Weight update using back propagation.

  • Forward pass of the second instance. calculate the error.

  • Weight update using back propagation. and so on... (or)

  • Forward pass of all instances one by one. (noting the error as vector)

  • Weight update using back propagation.

This video https://www.youtube.com/watch?v=OwdjRYUPngE is similar to the second process. Is it correct?

$\endgroup$
  • $\begingroup$ I think the video you watched is an example of batch training. Meaning that you activate an x amount of samples from your set, and you calculate the error over those samples and then change the weights. So you change the weights every x samples. However, if you are a newbie, you should first focus on online learning: activating 1 sample, then backpropagating and repeating that for every sample in the set. $\endgroup$ – Thomas W Jun 15 '17 at 14:17
0
$\begingroup$

Both are feasible. A generalization is to apply the modification of the weights after the presentation of n examples. If n=1, this is online training. If n=60 (60 in your case), this is batch training. In other cases, this is mini-batch training.
The main difference is not the computation complexity of the algorithm but the theoretical speed of convergence to an "optimal" set of weights. It is now generally admitted that online training is faster. Online training performs a stochastic approximation of the gradient descent method. The true gradient of the cost function over the whole training set is approximated by a gradient at each presentation of a training example.
Intuitively, with batch learning the weights are modified by an average of the gradient over the whole training set and averaging destroys information. While with stochastic (or mini-batch learning), each example has his word to say. One after the other. :)
This has been discussed here : https://stats.stackexchange.com/questions/49528/batch-gradient-descent-versus-stochastic-gradient-descent

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.