I am currently training some models using gradient accumulation since the model batches do not fit in GPU memory. Since I am using gradient accumulation, I had to tweak the training configuration a bit. There are two parameters that I tweaked: the batch size and the gradient accumulation steps. However, I am not sure about the effects of this modification, so I would like to fully understand what is the relationship between the gradient accumulation steps parameter and the batch size.

I know that when you accumulate the gradient you are just adding the gradient contributions for some steps before updating the weights. Normally, you would update the weights every time you compute the gradients (traditional approach):

$$w_{t+1} = w_t - \alpha \cdot \nabla_{w_t}loss$$

But when accumulating gradients you compute the gradients several times before updating the weights (being $N$ the number of gradient accumulation steps):

$$w_{t+1} = w_t - \alpha \cdot \sum_{0}^{N-1} \nabla_{w_t}loss$$

My question is: What is the relationship between the batch size $B$ and the gradient accumulation steps $N$?

By example: are the following configurations equivalent?

  • $B=8, N=1$: No gradient accumulation (accumulating every step), batch size of 8 since it fits in memory.
  • $B=2, N=4$: Gradient accumulation (accumulating every 4 steps), reduced batch size to 2 so it fits in memory.

My intuition is that they are but I am not sure. I am not sure either if I would have to modify the learning rate $\alpha$.


1 Answer 1


There isn't any explicit relation between the batch size and the gradient accumulation steps, except for the fact that gradient accumulation helps one to fit models with relatively larger batch sizes (typically in single-GPU setups) by cleverly avoiding memory issues. The core idea of gradient accumulation is to perform multiple backward passes using the same model parameters before updating them all at once for multiple batches. This is unlike the conventional manner, where the model parameters are updated once every batch-size number of samples. Therefore, finding the correct batch-size and accumulation steps is a design trade-off that has to be made based on two things: (i) how much increase in the batch-size can the GPU handle, and (ii) whether the gradient accumulation steps result in at least as much better performance than without accumulation.

As for your example configurations, there are the same in theory. But, there are a few important caveats that need to be addressed before proceeding with this intuition.

  1. Using Batch Normalization with gradient accumulation generally does not work well, simply because BatchNorm statistics cannot be accumulated. A better solution would be to use Group Normalization instead of BatchNorm.
  2. When performing a combined update in gradient accumulation, it must be ensured that the optimizer is not initialized to zero (i.e. optimizer.zero_grad()) for every backward update (i.e. loss.backward()). It is easy to include both statements in the same for loop while training which defeats the purpose of gradient accumulation.

Here are some interesting resources to find out more about it detail.

  1. Thomas Wolf's article on different ways of combating memory issues.
  2. Kaggle discussions on the effect on learning rate: here and here.
  3. A comprehensive answer on gradient accumulation in PyTorch.
  • $\begingroup$ Thank you for your answer. The caveats section was specially helpful. I was already aware of point (2) but point (1) was very ilustrative. As you hint, it seems it is better to move to multi GPU... I'll add it to the TODO list. $\endgroup$
    – JVGD
    Commented Jun 18, 2020 at 6:56
  • $\begingroup$ What happens if we have a variable batch-size during gradient accumulation? I used to normalize my gradients by the number of accumulation steps but now I've moved to a bucketing technique that results in lesser samples the longer the input sequence gets. $\endgroup$ Commented Jun 23, 2021 at 7:07
  • 1
    $\begingroup$ As I understand it, can accumulating gradients be used to simulate arbitrarily large batch sizes? If the gradients are just added for each minibatch, then shouldn't one be able to scale N (the number of iterations before accumulation) to be arbitrarily large? $\endgroup$
    – Dieblitzen
    Commented Feb 18, 2023 at 17:38
  • $\begingroup$ Don't you mean "multiple forward passes"? "Multiple backward passes before updating parameters" is a sentence that doesn't make any sense because a backward pass is updating parameters. $\endgroup$
    – Beyondo
    Commented Mar 10 at 14:49

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .