The choice of the batch size to be a power of 2 is not due the quality of predictions .

The larger the batch_size is - the better is the estimate of the gradient, but a noise can be beneficial to escape local minima. However, there is not much different say in the behavior for training with batch_size = 61 ,and batch_size = 64.

The actual reason, as explained here, is due to the alignment of virtual and physical processors. Usually, the number of physical processors is some multiple of power of two (number of CUDA streaming multiprocessors).

If the number of virtual processors is multiple of power of two, then each physical processor would be responsible for the same amount of virtual processors. Otherwise, some of them would stay idle.

So the reason for batches to be powers of 2 is about efficient utilization of GPU.

The choice of the batch size to be a power of 2 is not due the quality of predictions .

The larger the batch_size is - the better is the estimate of the gradient, but a noise can be beneficial to escape local minima. However, there won't be much difference in optimization procedure for batch_size=61 and batch_size=64, since the amount of stochasticity would be of the same order of magnitude.

The actual reason, as explained here, is due to the alignment of virtual and physical processors. Usually, the number of physical processors is some multiple of power of two (number of CUDA streaming multiprocessors).

If the number of virtual processors is multiple of power of two, then each physical processor would be responsible for the same amount of virtual processors. Otherwise, some of them would stay idle.

So the reason for batches to be powers of 2 is about efficient utilization of GPU.