Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
Why is a batch size needed to update the weights of a neural network?
According to that Youtube Video from 3B1B, the weights are updated by calculating the error between expectation and outcome of the neural net. Based on that, the chain rule is applied to calculate the new weights.
Following that logic, why would I pass a complete batch through the net? The first entries wouldn't have an impact on the weighting.
Do I need to define a batch size when I use backpropagation?
tl;dr: A batch size is the number of samples a network sees before updating its gradients. This number can range from a single sample to the whole training set. Empirically, there is a sweet spot in the range 1 to a few hundreds, where people experience the fastest training speeds. Check this article for more details.
If you have a small enough number of samples, you can let the network see all of the samples before updating its weights; this is called Gradient Descent. The benefit from this is that you guarantee that the weights will be updated in the direction that reduces the training loss for the whole dataset. The downside is that it is computationally expensive and in most cases infeasible for deep neural nets.
What is done in practice is that the network sees only a batch of the training data, instead of the whole dataset, before updating its weights. However, this technique does not guarantee that the network updates its weights in a way that will reduce the dataset's training loss; instead it reduces the batch's training loss, which might not the same thing. This adds noise to the training process, which can in some cases be a good thing, but requires the network to take too many steps to converge (this isn't a problem since each step is much faster).
What you're saying is essentially training the network each time on a single sample. This is formally called Stochastic Gradient Descent, however the term is used more broadly to include any case where the network is trained on a subset of the whole training set. The problem with this approach is that it adds too much noise to the training process, causing it to require a lot more steps to actually converge.
We basically distinguish between 3 forms of batch training:
$$Loss_{minibatch} = \sum_{m} l_m(\mathbf{W},t_m) \;\;\; with \;m \;\epsilon \; M$$
where M is a (random) subset of the whole dataset.
$$Loss_{batch} = \sum_{b} l_m(\mathbf{W},t_b) \;\;\; with \;b \;\epsilon \; B$$
where B is the whole dataset.
$$Loss_{stochastic} = l_i(\mathbf{W},t_i) $$
where i is a single sample from the whole dataset.
Here t is the target/label of a sample m,b,i and W are the network weights. The most common case today is usually minibatch training.
When we are training(updating the weights of the neural network to optimize towards a lower Loss) we take the derivative of this loss function with respect to the weights W. This will give us the gradient of the NN which tells us how much and in what direction we should update each weight.
$$ \nabla L = \frac{dLoss_{minibatch}}{d \mathbf{W}} = \frac{d\sum_m l_m(\mathbf{W},t_m)}{d \mathbf{W}} = \sum_m \frac{d l_m(\mathbf{W},t_m)}{d \mathbf{W}} = \sum_m \nabla l_m$$
As you can see here in the example of the minibatch case: the total gradient is the sum of the gradient of each sample in the minibatch. So why do you think the first elemnts do not have an impact on the weight update? Or do I understand you wrong?
