Should the weights of a neural network be updated after each example or at the end of the batch? Do I need a normalization factor in the second case?
The backpropagation step is generally used to compute the gradients and update the weights.
Let us say, you are implementing gradient descent, select the whole training batch, perform the forward propagation using the current set of neurons/weights to get the classification output. Then compute the loss/cost with respect to the actual labels, (Note: This step contains dividing the loss and divide it by the number of training example to get the cost function). Once you get the cost you backpropagate by computing the gradients at each layer with respect to cost from output to input and perform the weight updates.
If you are using mini-batch, you compute the cost for that mini-batch, divide by the number of examples in the mini-batch, then compute the gradients and perform the update in the backprop step.
When am I supposed to update my weights? After each forward-, and backpropagation; and or after each completed batch?
In your example, you should update the weights after each back propagation.
Also, you can back propagate the result by adding each error/loss for each prediction/example in a batch and then update the weights. pseudocode- for batch size 3:
w = backprop(Error(t1)+Error(t2)+Error(t3))
- Now what happen If you back propagate every error for every prediction/example in a batch and store them(without updating), and after completion of that batch update the weights by using stored calculated values, then there is a chance to stuck in the local minima also some research shows that it's actually slow training. research paper
note- all those techniques in 1,2,3 are different but 1 and 2 results same. Also in 2, adding errors of different training example makes a new graph of neural network and back propagating it means back propagating the resulted graph.