Should the weights of a neural network be updated after each example or at the end of the batch? Do I need a normalization factor in the second case?

  • $\begingroup$ Ideally you would use a small enough batch that uses the best of both worlds. In the first case, you have a lot of variance between the gradients in each sample, but, you have significantly more updates which could result in faster training, can cause catastrophic forgetting if your data isn't random. In the latter case you would either use the average, or sum the batch and multiply by a small value. The latter exhibits less variance but you need to be careful with the lr. $\endgroup$ – Dimitris Monroe Nov 4 '19 at 8:42

The backpropagation step is generally used to compute the gradients and update the weights.

Let us say, you are implementing gradient descent, select the whole training batch, perform the forward propagation using the current set of neurons/weights to get the classification output. Then compute the loss/cost with respect to the actual labels, (Note: This step contains dividing the loss and divide it by the number of training example to get the cost function). Once you get the cost you backpropagate by computing the gradients at each layer with respect to cost from output to input and perform the weight updates.

If you are using mini-batch, you compute the cost for that mini-batch, divide by the number of examples in the mini-batch, then compute the gradients and perform the update in the backprop step.

  • $\begingroup$ Okay so to summarize, you update the weights after each completed batch. This means the update frequency depend on the batch size. $\endgroup$ – Sebastian Nielsen Oct 20 '18 at 21:55
  • $\begingroup$ Yes, if you choose smaller batches then there will be a large number of updates per epoch. If you take the whole training data in one batch there will be just one update per epoch. $\endgroup$ – fireboy Oct 22 '18 at 4:07

When am I supposed to update my weights? After each forward-, and backpropagation; and or after each completed batch?

  1. In your example, you should update the weights after each back propagation.

  2. Also, you can back propagate the result by adding each error/loss for each prediction/example in a batch and then update the weights. pseudocode- for batch size 3:
    w = backprop(Error(t1)+Error(t2)+Error(t3)) update_wights(w*learning_rate)

  3. Now what happen If you back propagate every error for every prediction/example in a batch and store them(without updating), and after completion of that batch update the weights by using stored calculated values, then there is a chance to stuck in the local minima also some research shows that it's actually slow training. research paper

note- all those techniques in 1,2,3 are different but 1 and 2 results same. Also in 2, adding errors of different training example makes a new graph of neural network and back propagating it means back propagating the resulted graph.


Not the answer you're looking for? Browse other questions tagged or ask your own question.