1
$\begingroup$

In the stochastic gradient descent algorithm, the weight update happens for every training sample.

In the mini-batch gradient descent algorithm, the weight update happens for every batch of training samples.

In the batch gradient descent algorithm, the weight update happens for all samples in the training dataset.

I am confused with the procedure of training that happens in the mini-batch gradient descent algorithm. I am guessing one of the following two must be correct

  1. Passing each input individually at each layer and calculating the output. This happens for a number of training samples that are equal to batch size.

  2. Passing a batch of inputs at once at each layer and collecting the batch output at each layer.

Which of the above is true in general implementations of mini-batch gradient descent algorithms to train your neural networks?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

In the usual scenario, case 2 occurs. In the deep learning frameworks, Tensors have a special dimension (usually corresponding to the 0 axes) which numerates the example in the batch. Look for example in the PyTorch documentation of Conv2d or Tensorflow documentation of Conv2d. The same is true for any Layer - Linear, MultiheadAttention, RNN.

All samples from the batch are processed at once, as the integral entity. Most operations process each sample from the batch independently, without a combination of features from $i^{th}$ and $j^{th}$ sample. Linear layer, Convolution doesn't construct linear combinations of inputs, corresponding to different samples.
However, there is an exception - the BatchNormalization Layer which subtracts the mean and divides by the standard deviation.

However, one may want to work with large batches in order to have a less noisy and more precise estimate of the gradient at each step, but the memory usage increases linearly with the batch size since one has to allocate batch size times the memory, that is required for propagating a single sample through the network. In case, the computational resources do not allow for storing such a large batch in memory, one can pass a part of a large batch or even a single example, and only then aggregate the results.

This situation corresponds to the case 1. Such functionality is implemented in the PyTorch-Lightning library .

Improve this answer
$\endgroup$
0
$\begingroup$

What happens in mini-batches is not very different from the way updates are made in batch gradient descent, only the number of samples is different. In mini-batch, you process all the data in the batch, and the update happens after that. It is detailed in this video after 6:11.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Which one is correct #1 or #2? $\endgroup$
    – hanugm
    Sep 27, 2021 at 9:17
  • 1
    $\begingroup$ I forgot about batch normalization layer, as mention by @spiridon_the_sun_rotator, which requires the process defined in option 2 given above to define a mean and variance at each layer, over each batch. $\endgroup$
    – serali
    Sep 27, 2021 at 10:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .