What is a "batch" in batch normalization?

Question

I'm working on an example of CNN with the MNIST hand-written numbers dataset. Currently I've got convolution -> pool -> dense -> dense, and for the optimiser I'm using Mini-Batch Gradient Descent with a batch size of 32.

Now this concept of batch normalization is being introduced. We are supposed to take a "batch" after or before a layer, and normalize it by subtracting its mean, and dividing by its standard deviation.

So what is a "batch"? If I feed a sample into a 32 kernel conv layer, I get 32 feature maps.

• Is each feature map a "batch"?
• Are the 32 feature maps the "batch"?

Or, if I'm doing Mini-Batch Gradient Descent with a batch size of 64,

• Are 64 sets of 32 feature maps the "batch"? So in other words, the batch from Mini-Batch Gradient Descent, is the same as the "batch" from batch-optimization?

Or is a "batch" something else that I've missed?

Answer 1

The "batch" is same as in mini-batch gradient descent. The mean in batch-norm here would be the average of each feature map in your batch (in your case either 32 or 64 depending on which you use)

generally batch is used quite consistently in ML right now, where it refers to the inputs you send in together for forward/backward pass.