0
$\begingroup$

I am currently running a program with a batch size of 17 instead of batch size 32. The benchmark results are obtained at a batch size of 32 with the number of epochs 700.

Now I am running with batch size 17 with unchanged number epochs. So I am interested to know whether there is any relationship between the batch size and the number of epochs in general.

Do I need to increase the number of epochs? Or is it entirely dependent on the program?

$\endgroup$
0
$\begingroup$

Smaller batch size means the model is updated more often. So, it takes longer to complete each epoch. Also, if the batch size is too small, each update is done without "seeing" all the data - the batch itself might not be a good representative of the dataset. So, there might be too much "wiggling", which makes it harder to get real minimum. Larger batches will get you "near" minimum for quickly, as they have larger and step sizes.

As you only asked the relation between batch size and epochs, above is the answer. But in practice, one hardly uses very large batches as it doesn't get very close to the minimum for the same reason they reach there faster - once you are there, you want to have smaller steps to get very close to the minimum. Smaller batches might take longer but once they are there, their wiggly nature becomes a strength, and gets them closer to the minimum.

For a more in depth discussion and some references, see this post.

And batch sizes are usually picked to be a power of 2, so I would go for 16 instead of 17 if I were you. Check this discussion for the reason for this.

$\endgroup$
3
  • $\begingroup$ "Smaller batch size means the model is updated more often. So, it takes longer to complete each epoch", but if you use fewer examples at each update, wouldn't that make the update faster? So, wouldn't that smaller times for updating compensate the higher number of updates? I don't know exactly, but that statement/conclusion may not be correct. Note that I am not talking about convergence here, I'm just talking about time complexity. $\endgroup$
    – nbro
    Sep 25 at 14:11
  • $\begingroup$ Well, still each time the update is made, the entire model is updated, that is the issue. Small batch or large batch, for each epoch entire dataset is used once in the calculation. Assuming total time for the calculation excluding updates is the same for both, the difference comes from the number of updates. And the number of times an update is made is higher for small batches. $\endgroup$
    – serali
    Sep 25 at 14:31
  • $\begingroup$ @nbro Sorry, I guess I did not address your comment directly but I don't know why small samples would mean shorter update time. Even we assume the changes are small in small batches, the mathematical operation made on each cell is the same, and should take the same amount of time. $\endgroup$
    – serali
    Sep 25 at 14:43
0
$\begingroup$

The smaller the batch_size is, the larger number of batches is processed per one epoch.

On one hand, since one makes more steps per epoch, one can think, that less epochs are required to achieve the same level of accuracy.

On the other side, smaller batch size leads to more noisy and stochastic estimates of the gradient, therefore, convergence would not be as steady most likely.

I think it is difficult to give a definite answer about the exact relation on the number of epochs - since say, to achieve a certain level of accuracy use of small batch may be more beneficial since it allows for more exploration and is more likely to escape from local minima and saddles, but when one reaches the approximation limit of the network and is in the vicinity of the good optimum - large batch would descend better to the extremum.

Good strategy usually is to start from smaller batches to find wide and plain minima, which are better from the generalization point of view, and then increase batch size for steadier convergence.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.