What is the relationship between gradient accumulation and batch size?

Question

I am currently training some models using gradient accumulation since the model batches do not fit in GPU memory. Since I am using gradient accumulation, I had to tweak the training configuration a bit. There are two parameters that I tweaked: the batch size and the gradient accumulation steps. However, I am not sure about the effects of this modification, so I would like to fully understand what is the relationship between the gradient accumulation steps parameter and the batch size.

I know that when you accumulate the gradient you are just adding the gradient contributions for some steps before updating the weights. Normally, you would update the weights every time you compute the gradients (traditional approach):

$$w_{t+1} = w_t - \alpha \cdot \nabla_{w_t}loss$$

But when accumulating gradients you compute the gradients several times before updating the weights (being $N$ the number of gradient accumulation steps):

$$w_{t+1} = w_t - \alpha \cdot \sum_{0}^{N-1} \nabla_{w_t}loss$$

My question is: What is the relationship between the batch size $B$ and the gradient accumulation steps $N$?

By example: are the following configurations equivalent?

• $B=8, N=1$: No gradient accumulation (accumulating every step), batch size of 8 since it fits in memory.
• $B=2, N=4$: Gradient accumulation (accumulating every 4 steps), reduced batch size to 2 so it fits in memory.

My intuition is that they are but I am not sure. I am not sure either if I would have to modify the learning rate $\alpha$.

Answer 1

There isn't any explicit relation between the batch size and the gradient accumulation steps, except for the fact that gradient accumulation helps one to fit models with relatively larger batch sizes (typically in single-GPU setups) by cleverly avoiding memory issues. The core idea of gradient accumulation is to perform multiple backward passes using the same model parameters before updating them all at once for multiple batches. This is unlike the conventional manner, where the model parameters are updated once every batch-size number of samples. Therefore, finding the correct batch-size and accumulation steps is a design trade-off that has to be made based on two things: (i) how much increase in the batch-size can the GPU handle, and (ii) whether the gradient accumulation steps result in at least as much better performance than without accumulation.

As for your example configurations, there are the same in theory. But, there are a few important caveats that need to be addressed before proceeding with this intuition.

1. Using Batch Normalization with gradient accumulation generally does not work well, simply because BatchNorm statistics cannot be accumulated. A better solution would be to use Group Normalization instead of BatchNorm.
2. When performing a combined update in gradient accumulation, it must be ensured that the optimizer is not initialized to zero (i.e. optimizer.zero_grad()) for every backward update (i.e. loss.backward()). It is easy to include both statements in the same for loop while training which defeats the purpose of gradient accumulation.

Here are some interesting resources to find out more about it detail.

1. Thomas Wolf's article on different ways of combating memory issues.
2. Kaggle discussions on the effect on learning rate: here and here.
3. A comprehensive answer on gradient accumulation in PyTorch.