24
$\begingroup$

Batch size is a term used in machine learning and refers to the number of training examples utilised in one iteration. The batch size can be one of three options:

  1. batch mode: where the batch size is equal to the total dataset thus making the iteration and epoch values equivalent
  2. mini-batch mode: where the batch size is greater than one but less than the total dataset size. Usually, a number that can be divided into the total dataset size.
  3. stochastic mode: where the batch size is equal to one. Therefore the gradient and the neural network parameters are updated after each sample.

How do I choose the optimal batch size, for a given task, neural network or optimization problem?

If you hypothetically didn't have to worry about computational issues, what would the optimal batch size be?

$\endgroup$

3 Answers 3

7
$\begingroup$

Here are a few guidelines, inspired by the deep learning specialization course, to choose the size of the mini-batch:

  • If you have a small training set, use batch gradient descent (m < 200)

In practice:

  • Batch mode: long iteration times
  • Mini-batch mode: faster learning
  • Stochastic mode: lose speed up from vectorization

The typically mini-batch sizes are 64, 128, 256 or 512.

And, in the end, make sure the minibatch fits in the CPU/GPU.

Have also a look at the paper Practical Recommendations for Gradient-Based Training of Deep Architectures (2012) by Yoshua Bengio.

$\endgroup$
3
  • 3
    $\begingroup$ How does one normally check if the minibatch fits the CPU? and the GPU? $\endgroup$
    – gota
    Nov 15, 2020 at 19:57
  • $\begingroup$ you know that training a deep learning model that has a lot of parameters has a relation to CPU obviously, for practical guidelines especially from an academic perspective, there is that tradeoff between accuracy and computational resources. But if you find something very useful please share it here :) $\endgroup$ Oct 12, 2021 at 15:56
  • $\begingroup$ @gota You can calculate this in advance, but the easiest way to discover what fits in GPU memory is to increase the batch size until training fails from lack of memory. This should happen pretty quickly. $\endgroup$ Sep 18 at 12:56
6
$\begingroup$

From the blog A Gentle Introduction to Mini-Batch Gradient Descent and How to Configure Batch Size (2017) by Jason Brownlee.

How to Configure Mini-Batch Gradient Descent

Mini-batch gradient descent is the recommended variant of gradient descent for most applications, especially in deep learning.

Mini-batch sizes, commonly called “batch sizes” for brevity, are often tuned to an aspect of the computational architecture on which the implementation is being executed. Such as a power of two that fits the memory requirements of the GPU or CPU hardware like 32, 64, 128, 256, and so on.

Batch size is a slider on the learning process.

  • Small values give a learning process that converges quickly at the cost of noise in the training process.
  • Large values give a learning process that converges slowly with accurate estimates of the error gradient.

Tip 1: A good default for batch size might be 32.

$\endgroup$
1
  • $\begingroup$ The main content in this answer was completely copied from another source. We highly discourage this type of answer, but we prefer that users give answers in their own words, unless the quote is particularly relevant, but, even in that case, we expect users to provide more than just the quote. $\endgroup$
    – nbro
    Oct 31, 2020 at 10:05
4
$\begingroup$

The batch size can also have a significant impact on your model’s performance and the training time. In general, the optimal batch size will be lower than 32. In April 2018, Yann Lecun even tweeted:

Training with large minibatches is bad for your health. More importantly, it's bad for your test error. Friends dont let friends use minibatches larger than 32.

A small batch size ensures that each training iteration is very fast, and although a large batch size will give a more precise estimate of the gradients, in practice this does not matter much since the optimization landscape is quite complex and the direction of the true gradients do not point precisely in the direction of the optimum.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .