The choice of the batch size to be a power of 2 is not due the quality of predictions .
The larger the batch_size is - the better is the estimate of the gradient, but a noise can be beneficial to escape local minima. However, there won't be much difference in optimization procedure for
batch_size=64, since the amount of stochasticity would be of the same order of magnitude.
The actual reason, as explained here, is due to the alignment of virtual and physical processors. Usually, the number of physical processors is some multiple of power of two (number of
CUDA streaming multiprocessors).
If the number of virtual processors is multiple of power of two, then each physical processor would be responsible for the same amount of virtual processors. Otherwise, some of them would stay idle.
So the reason for batches to be powers of 2 is about efficient utilization of GPU.