4
$\begingroup$

Batch size is a term used in machine learning and refers to the number of training examples utilised in one iteration. The batch size can be one of three options:

  1. batch mode: where the batch size is equal to the total dataset thus making the iteration and epoch values equivalent
  2. mini-batch mode: where the batch size is greater than one but less than the total dataset size. Usually, a number that can be divided into the total dataset size.
  3. stochastic mode: where the batch size is equal to one. Therefore the gradient and the neural network parameters are updated after each sample.

How do I choose the optimal batch size, for a given task, neural network or optimization problem?

If you hypothetically didn't have to worry about computational issues, what would the optimal batch size be?

$\endgroup$
2
$\begingroup$

From this awesome blog


How to Configure Mini-Batch Gradient Descent

Mini-batch gradient descent is the recommended variant of gradient descent for most applications, especially in deep learning.

Mini-batch sizes, commonly called “batch sizes” for brevity, are often tuned to an aspect of the computational architecture on which the implementation is being executed. Such as a power of two that fits the memory requirements of the GPU or CPU hardware like 32, 64, 128, 256, and so on.

Batch size is a slider on the learning process.

  • Small values give a learning process that converges quickly at the cost of noise in the training process.
  • Large values give a learning process that converges slowly with accurate estimates of the error gradient.

A good default for batch size might be 32

$\endgroup$
2
$\begingroup$

Here are a few guidelines, inspired by the deep learning specialization course, to choose the size of the mini-batch:

  • If you have a small training set, use batch gradient descent (m < 200)

In practice:

  • Batch mode: long iteration times
  • Mini-batch mode: faster learning
  • Stochastic mode: lose speed up from vectorization

The typically mini-batch sizes are 64, 128, 256 or 512.

And, in the end, make sure the minibatch fits in the CPU/GPU.

Have also a look at the paper Practical Recommendations for Gradient-Based Training of Deep Architectures (2012) by Yoshua Bengio.

$\endgroup$
0
$\begingroup$

How many examples are passed through an artificial network prior to calculating loss and back propagating the correction signal determined according to gradient descent is one part of the convergence strategy that we currently call learning. Whether a stochastic element is introduced into the gradient descent process is another. The two should not be linked so intimately together.

You can use stochastic gradient descent in batch learning and you can train without batching and without deterministic gradient descent.

Batch size means different things for the two different batch modes too. One can have a batch size of 500 because that is the training set size, but one can have a mini batch size of 5,000 because the training set size is 500,000.

There are four choices to make in this decision space.

  • How much of the total data available will be used for training, testing, and validation
  • Whether to not batch at all, mini-batch, or all in one batch
  • Whether to use stochastic gradient descent or deterministic
  • In the case of mini-batch ONLY, how large is the mini batch

Some of the frameworks themselves suggest starting values or heuristics for determining them. Academic papers usually document the choices they found that worked best. One reasonable heuristic for mini-batch training and test runs is to use the square root of the size of the data set drawn.

For example, if there are 100,000 examples and 10% is randomly drawn for training, a realistic mini batch size might be 100. There would be 100 batches of that size from the 10,000 training examples. The test run would mirror that, drawing the samples using the same random mechanism.

If you hypothetically don't have to worry about computational issues, which is almost never the case, then the heuristic used in the above example would be reasonable.

$\endgroup$
  • 1
    $\begingroup$ We use the terminology "stochastic gradient descent" as soon as we use batch sizes smaller than the full training data. It really doesn't refer to any other addition of some form of stochasticity, just the stochasticity in the gradient estimates resulting from the fact the gradients are estimated on a sample of the training data. There is no such thing as "deterministic gradient descent" with mini-batches, those are not separate choices $\endgroup$ – Dennis Soemers Oct 23 '18 at 15:06
-1
$\begingroup$

The divisions in batching of example data to train artificial networks have little to do with batch size, per se, but rather whether batching is used at all and whether there is more than one batch. The names they were given are based on the existence of a design approach in the popular Python framework, TensorFlow, where they are called modes.

KDnuggets August 2016 News has a tutorial on the TensorFlow modes, and page two of part two of that article has three instructive images that may assist in understanding, reproduced here in case the domain name or path changes.

batch mode

mini-batch mode

stochastic mode

You can see why the designers of TensorFlow call them modes. They are not drastically different approaches when one considers the overall mechanics of training process. And that becomes clear in these control flow diagrams.

In synopsis the mode names in TensorFlow correspond to three ways to group input examples.

  • Batch mode = All training examples are processed in a single iterative descent process.
  • Mini-batch mode = Training examples are placed in more than one group and each group is processed, one at a time in sequence, in an iterative descent process.
  • Stochastic mode = Each training example is used in sequence in a single iterative descent process, which almost without exception produces an inherently noisy descent, so they call it stochastic.

Note that stochastic elements can be deliberately injected into batch and mini-batch training operations, so using single examples in sequential training iterations is not the only way to get the benefit of partially stochastic searching.

More directly related to the core of the question, it is only mini-batch mode that requires thinking about batch size. If one wants to have the batch size be close to the number of batches, one can find a whole number pair of factors that multiply to equal the total number of training examples as close to the square root of the number of training examples as possible.

That is a common approach, but one may wish, for some specific reason to have a larger batch size than number of batches or vise versa. Those choices are usually related to speed of convergence.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.