I'm working on an example of CNN with the MNIST hand-written numbers dataset. Currently I've got convolution -> pool -> dense -> dense, and for the optimiser I'm using Mini-Batch Gradient Descent with a batch size of 32.
Now this concept of batch normalization is being introduced. We are supposed to take a "batch" after or before a layer, and normalize it by subtracting its mean, and dividing by its standard deviation.
So what is a "batch"? If I feed a sample into a 32 kernel conv layer, I get 32 feature maps.
- Is each feature map a "batch"?
- Are the 32 feature maps the "batch"?
Or, if I'm doing Mini-Batch Gradient Descent with a batch size of 64,
- Are 64 sets of 32 feature maps the "batch"? So in other words, the batch from Mini-Batch Gradient Descent, is the same as the "batch" from batch-optimization?
Or is a "batch" something else that I've missed?