Batch size is a term used in machine learning and refers to the number of training examples utilised in one iteration. The batch size can be one of three options:
- batch mode: where the batch size is equal to the total dataset thus making the iteration and epoch values equivalent
- mini-batch mode: where the batch size is greater than one but less than the total dataset size. Usually, a number that can be divided into the total dataset size.
- stochastic mode: where the batch size is equal to one. Therefore the gradient and the neural network parameters are updated after each sample.
How do I choose the optimal batch size, for a given task, neural network or optimization problem?
If you hypothetically didn't have to worry about computational issues, what would the optimal batch size be?