I am confused among the following in selecting the batch size for my model.
#1: powers of 2
I generally see that batch sizes are in powers of two: 32, 64, 128, 256.
#2: maximum GPU
Suppose my GPU allows a maximum batch size of 61. And it is not a power of two.
Which one should I apt? Is there anything like the powers of 2 will give relatively good results?
The choice of the batch size to be a power of 2 is not due the quality of predictions .
The larger the batch_size is - the better is the estimate of the gradient, but a noise can be beneficial to escape local minima. However, there won't be much difference in optimization procedure for batch_size=61 and batch_size=64, since the amount of stochasticity would be of the same order of magnitude.
The actual reason, as explained here, is due to the alignment of virtual and physical processors. Usually, the number of physical processors is some multiple of power of two (number of CUDA streaming multiprocessors).
If the number of virtual processors is multiple of power of two, then each physical processor would be responsible for the same amount of virtual processors. Otherwise, some of them would stay idle.
So the reason for batches to be powers of 2 is about efficient utilization of GPU.
The main reason to use powers of 2 is in the way existing hardware and software are made, there isn't any purely mathematical reason. CPUs, GPUs, memories, and internal buses all use a size that's the power of 2 since that's the most efficient way to address them.
