# How to take the optimal batch_size for training a model?

I have an image dataset, which is composed of 113695 images for training and 28424 images for validation. Now, when I use ImageDataGenerator and flow_from_dataframe, it as the parameter batch_size.

How can I take the correct number for batch_size because both numbers cannot be divided by the same number? Should I need to drop four images in the validation data to make them batch_size of 5? Or is there another way?

• Note that programming issues are off-topic here. I am not completely sure whether this is only a programming issue or not, so I will leave it open for now. Can you clarify if this is a programming issue or not? Please, read our on-topic page: ai.stackexchange.com/help/on-topic.
– nbro
Aug 21 '20 at 11:15

In a nutshell:

• A single batch (that is all your data in one batch) will result in a smooth trajectory on the loss surface. The drawback is that all your data might not fit into your memory. Which is highly likely for ~100k images.

• One image per batch (batch size = no. examples) will result in a more stochastic trajectory since the gradients are calculated on a single example. Advantages are of computational nature and faster training time.

The middle way is to choose the batch size in such a way that your batch fits into memory and gradients behave less 'noisy'. To be honest there is no 'golden' number, personally I like to choose powers of two.

Don't worry that your data is not divisible by the batch size. Libraries will take care about that internally, the last batch will just be a smaller than the defined batch size ($$N \text{ mod } b$$).

From Andrew lesson on Coursera, batch_size should be the power of 2, ex: 512, 1024, 2048. It will faster for training.

And you don't need to drop your last images to batch_size of 5 for example. The library likes Tensorflow or Pytorch, the last batch_size will be number_training_images % 5 which 5 is your batch_size.

Last but not least, batch_size need to fit your memory training (CPU or GPU). You can try several large batch_size to know which value is not out of memory. The smaller number_mini_batch = number_training_image//batch_size + 1, the faster for training time.