# In mini-batch gradient descent, do we pass each input in the batch individually or all inputs at the same time through the layer?

In the stochastic gradient descent algorithm, the weight update happens for every training sample.

In the mini-batch gradient descent algorithm, the weight update happens for every batch of training samples.

In the batch gradient descent algorithm, the weight update happens for all samples in the training dataset.

I am confused with the procedure of training that happens in the mini-batch gradient descent algorithm. I am guessing one of the following two must be correct

1. Passing each input individually at each layer and calculating the output. This happens for a number of training samples that are equal to batch size.

2. Passing a batch of inputs at once at each layer and collecting the batch output at each layer.

Which of the above is true in general implementations of mini-batch gradient descent algorithms to train your neural networks?

In the usual scenario, case 2 occurs. In the deep learning frameworks, Tensors have special dimension (usually corresponding to the 0 axis) which numerates the example in the batch. Look for example in the PyTorch documentation of Conv2d or Tensorflow documentation of Conv2d. The same is true for any Layer - Linear, MultiheadAttention, RNN.

All samples from the batch are processed at once, as the integral entity. Most operations process each sample from the batch independently, without a combination of features from $$i^{th}$$ and $$j^{th}$$ sample. Linear layer, Convolution doesn't construct linear combinations of inputs, corresponding to different samples.
However, there is an exception - the BatchNormalization Layer which subtracts the mean and divides by the standard deviation.

However, one may want to work with large batches in order to have a less noisy and more precise estimate of the gradient at each step, but the memory usage increases linearly with the batch size, since one has to allocate batch size times the memory, that is required for propagating a single sample thorugh the network. In case, the computational resources do not allow for storing such large batch in a memory, one can pass a part of large batch or even single example, and only then aggregate the results.

This situation corresponds to the case 1. Such functionality is implemented in the PyTorch-Lightning library .

What happens in mini-batches is not very different from the way updates are made in batch gradient descent, only the number of samples is different. In mini-batch, you process all the data in the batch, and the update happens after that. It is detailed in this video after 6:11.

• Which one is correct #1 or #2? Sep 27 at 9:17
• I forgot about batch normalization layer, as mention by @spiridon_the_sun_rotator, which requires the process defined in option 2 given above to define a mean and variance at each layer, over each batch. Sep 27 at 10:19