0
$\begingroup$

I am confused among the following in selecting the batch size for my model.

#1: powers of 2

I generally see that batch sizes are in powers of two: 32, 64, 128, 256.

#2: maximum GPU

Suppose my GPU allows a maximum batch size of 61. And it is not a power of two.

Which one should I apt? Is there anything like the powers of 2 will give relatively good results?

$\endgroup$
9
  • 1
    $\begingroup$ It seems that this question is a duplicate of this, which, however, seems to be more than one question, so it might be closed as too broad. I would recommend that you search a little bit on the site before asking a question. Many questions have already been asked, but, of course, the answers may not be good and even the questions may be poor, and can be deleted. If you think they are poor, you can ask the question again and state that. $\endgroup$
    – nbro
    Dec 27 '21 at 9:11
  • $\begingroup$ @nbro Yeah, I missed the question because of tags assigned for that question. I think it is related but not duplicate. The intention of this question is to select which one: either #1 or #2, if can't both. $\endgroup$
    – hanugm
    Dec 27 '21 at 9:59
  • $\begingroup$ What do you mean by select #1 or #2? #1 is the batch size, #2 is the GPU, so what does that mean? I think they are exact duplicates. Here, it seems that you're asking why do we use powers of 2. If that's not the question, then the answer below doesn't answer your question. Your question in the title is "Is it true that batch size of form $2^k$ gives better results?", which seems to be the same as "why do we use powers of 2?" $\endgroup$
    – nbro
    Dec 27 '21 at 10:02
  • $\begingroup$ I mean, suppose my GPU allows at most 61 as batch-size. Then should I opt for 32 or 61 is the question. @nbro $\endgroup$
    – hanugm
    Dec 27 '21 at 10:04
  • 1
    $\begingroup$ But maybe it's not worth it, if you ask the OP below to address that question too. $\endgroup$
    – nbro
    Dec 27 '21 at 10:11
4
$\begingroup$

The choice of the batch size to be a power of 2 is not due the quality of predictions .

The larger the batch_size is - the better is the estimate of the gradient, but a noise can be beneficial to escape local minima. However, there won't be much difference in optimization procedure for batch_size=61 and batch_size=64, since the amount of stochasticity would be of the same order of magnitude.

The actual reason, as explained here, is due to the alignment of virtual and physical processors. Usually, the number of physical processors is some multiple of power of two (number of CUDA streaming multiprocessors).

If the number of virtual processors is multiple of power of two, then each physical processor would be responsible for the same amount of virtual processors. Otherwise, some of them would stay idle.

So the reason for batches to be powers of 2 is about efficient utilization of GPU.

$\endgroup$
1
  • $\begingroup$ When you write "However, there is not much different say in the behavior for training with", I would write "For the purposes of optimization with SGD, there might not be much different in the behavior for training with say .... However, this is in theory. In practice, GPUs might have memory whose size is a power of 2..." etc $\endgroup$
    – nbro
    Dec 27 '21 at 9:22
1
$\begingroup$

The main reason to use powers of 2 is in the way existing hardware and software are made, there isn't any purely mathematical reason. CPUs, GPUs, memories, and internal buses all use a size that's the power of 2 since that's the most efficient way to address them.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.