# Effect of batch size and number of GPUs on model accuracy

I have a data set which was split using a fixed random seed and I am going to use 80% of data for training and rest on validation. Here are my GPU and batch size configurations

• use 64 batch size with one GTX 1080Ti
• use 128 batch size with two GTX 1080Ti
• use 256 batch size with four GTX 1080Ti

All other hyper-parameters such as lr, opt, loss, etc., are fixed. Notice the linearity between the batch size and the number of GPUs.

Will I get the same accuracy for those three experiments? Why and why not?

This should make a difference, but how big is the difference heavily depends on your task. However generally speaking, a smaller batch size will have a lower speed if counted in sample/minutes, but have a higher speed in batch/minutes. If the batch size is too small the batch/minute will be very low and therefore decreasing training speed severely. However a batch size too small(for example 1) will make the model hard to generalize and also slower to converge. Source

This slide is a great demonstration of how batch size affects training. As you can see from the diagram, when you have a small batch size the route to convergence will be ragged and not direct. This is because the model may train on an outlier and have it's performance decrease before fitting again. Of course this is an edge case and you would never train a model with 1 batch size. However on the other hand with a batch size too large, your model will take too long per iteration. With at least a decent batch size (like 16+) the amount of iterations needed to train the model is similar, so a larger batch size is not going to help a lot. The performance is not going to vary a lot.

In your case, the accuracy will make a difference but only minimally. Whilst writing this answer, I have ran a few test on batch size effect on performance and time, and here is the results. (Results to be added for 1 batch size)

Batch size 256 Time required 98.50849771499634s : 0.9414
Batch size 128 Time required 108.53689193725586s : 0.9668
Batch size 64 Time required 129.92272853851318s : 0.9776
Batch size 32 Time required 162.13709354400635s : 0.9844
Batch size 16 Time required 224.82269191741943s : 0.9854
Batch size 8 Time required 351.2729814052582s : 0.9861
Batch size 4 Time required 514.2667407989502s : 0.9862
Batch size 2 Time required 829.1623721122742s : 0.9869


You can test out yourself here.

As you can see, accuracy increase with the batch size decreasing and this is because a higher batch size means it will be trained on less iterations. 2x batch size = half the iterations so this is expected. The time required have risen exponentially, but batch size of 32 or below don't seems to make a large difference in time taken. The accuracy seems to be normal as half the iterations are trained with double the batch size.

In your case I would actually recommend you stick with 64 batch size even for 4 GPU. In the case of multi-gpu, the rule of thumb will be using at least 16 or so batch size per GPU, as if you are using 8 or 4 batch size the GPU cannot be completely utilized to train the model.

For multiple GPU accuracy impact, there is virtually no impact on accuracy as far as I know. Multiple GPU just take it's own slice of batches and computes it and feed the output to one GPU to calculate the loss, and the operations done across the GPUs are the same operations done on 1 GPU, so it shouldn't matter.

EDIT: For multiple GPU, there might me a slight difference due to percision error as told by @Posi2 . Please see here. Thanks @Posi2

## Conclusion

Batch size don't matter to performance too much as long as you set a reasonable batch size (16+) and keep the iterations not epochs the same. However training time will be affected and for multi-gpu you should use the minimum batch size for each GPU that will utilize 100% of the GPU to train. 16 per GPU is quite good.

• Yeah up to point.Slight difference in the accuracy of the model due to the number of GPUs because of floating-point approximation.Have a look stackoverflow.com/questions/43845644/… – Posi2 Jan 9 '20 at 11:43
• Thanks. Will add this to my answer. Thanks very much – Clement Hui Jan 9 '20 at 11:46
• Thanks very much @Posi2 – Clement Hui Jan 9 '20 at 11:51
• @ClementHui And each model saved from above three will have almost same inference accuracy, right? – bit_scientist Jan 10 '20 at 0:31
• They will have similar accuracy, give or take maybe 3% – Clement Hui Jan 10 '20 at 0:34

No. Different batch sizes mean different gradients (check stochastic gradient descent concept you will get how loss calculated) are calculated in each step, and thus the gradient descent will likely end up in different places in parameter space.

In addition, how this is actually parallelized might make a difference, including the order of operations and converting between FP precision.

Additional check resources:issue of multi gpus