The divisions in batching of example data to train artificial networks have little to do with batch size, per se, but rather whether batching is used at all and whether there is more than one batch. The names they were given are based on the existence of a design approach in the popular Python framework, TensorFlow, where they are called modes.
KDnuggets August 2016 News has a tutorial on the TensorFlow modes, and page two of part two of that article has three instructive images that may assist in understanding, reproduced here in case the domain name or path changes.
You can see why the designers of TensorFlow call them modes. They are not drastically different approaches when one considers the overall mechanics of training process. And that becomes clear in these control flow diagrams.
In synopsis the mode names in TensorFlow correspond to three ways to group input examples.
- Batch mode = All training examples are processed in a single iterative descent process.
- Mini-batch mode = Training examples are placed in more than one group and each group is processed, one at a time in sequence, in an iterative descent process.
- Stochastic mode = Each training example is used in sequence in a single iterative descent process, which almost without exception produces an inherently noisy descent, so they call it stochastic.
Note that stochastic elements can be deliberately injected into batch and mini-batch training operations, so using single examples in sequential training iterations is not the only way to get the benefit of partially stochastic searching.
More directly related to the core of the question, it is only mini-batch mode that requires thinking about batch size. If one wants to have the batch size be close to the number of batches, one can find a whole number pair of factors that multiply to equal the total number of training examples as close to the square root of the number of training examples as possible.
That is a common approach, but one may wish, for some specific reason to have a larger batch size than number of batches or vise versa. Those choices are usually related to speed of convergence.