We basically distinguish between 3 forms of batch training:
$$Loss_{minibatch} = \sum_{m} l_m(\mathbf{W},t_m) \;\;\; with \;m \;\epsilon \; M$$
where M
is a (random) subset of the whole dataset.
$$Loss_{batch} = \sum_{b} l_m(\mathbf{W},t_b) \;\;\; with \;b \;\epsilon \; B$$
where B
is the whole dataset.
$$Loss_{stochastic} = l_i(\mathbf{W},t_i) $$
where i
is a single sample from the whole dataset.
Here t
is the target/label of a sample m,b,i
and W
are the network weights. The most common case today is usually minibatch training.
When we are training(updating the weights of the neural network to optimize towards a lower Loss) we take the derivative of this loss function with respect to the weights W. This will give us the gradient of the NN which tells us how much and in what direction we should update each weight.
$$ \nabla L = \frac{dLoss_{minibatch}}{d \mathbf{W}} = \frac{d\sum_m l_m(\mathbf{W},t_m)}{d \mathbf{W}} = \sum_m \frac{d l_m(\mathbf{W},t_m)}{d \mathbf{W}} = \sum_m \nabla l_m$$
As you can see here in the example of the minibatch case: the total gradient is the sum of the gradient of each sample in the minibatch. So why do you think the first elemnts do not have an impact on the weight update? Or do I understand you wrong?