3
$\begingroup$

I'm working on an example of CNN with the MNIST hand-written numbers dataset. Currently I've got convolution -> pool -> dense -> dense, and for the optimiser I'm using Mini-Batch Gradient Descent with a batch size of 32.

Now this concept of batch normalization is being introduced. We are supposed to take a "batch" after or before a layer, and normalize it by subtracting its mean, and dividing by its standard deviation.

So what is a "batch"? If I feed a sample into a 32 kernel conv layer, I get 32 feature maps.

  • Is each feature map a "batch"?
  • Are the 32 feature maps the "batch"?

Or, if I'm doing Mini-Batch Gradient Descent with a batch size of 64,

  • Are 64 sets of 32 feature maps the "batch"? So in other words, the batch from Mini-Batch Gradient Descent, is the same as the "batch" from batch-optimization?

Or is a "batch" something else that I've missed?

$\endgroup$
3
$\begingroup$

The "batch" is same as in mini-batch gradient descent. The mean in batch-norm here would be the average of each feature map in your batch (in your case either 32 or 64 depending on which you use)

generally batch is used quite consistently in ML right now, where it refers to the inputs you send in together for forward/backward pass.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks! "The mean in batch-norm here would be the average of each feature map in your batch (in your case either 32 or 64 depending on which you use)." Okay so in the particular example I listed above, the batch would consist of 32x64=2048 feature maps. And although I haven't listed it, the feature maps are 26x26x1. So we find the mean and stdev across all 2048x26x26=73728 data points then normalise all data points according to that. Right? $\endgroup$ – Alexander Soare Dec 27 '19 at 17:18
  • $\begingroup$ @AlexanderSoare is 32 or 64 your batch size? or is 64 the channel size? $\endgroup$ – mshlis Dec 27 '19 at 17:21
  • $\begingroup$ The batch size is 64. There are 32 kernels in the conv layer so I get 32 channels out. $\endgroup$ – Alexander Soare Dec 27 '19 at 17:27
  • $\begingroup$ @AlexanderSoare your adding 64 elements together (where each element is a 26x26x32 tensor in your case) $\endgroup$ – mshlis Dec 27 '19 at 17:45
  • $\begingroup$ Thanks! Okay so then just to be super clear, and thanks again for your time, That means we are essentially taking 32x64x26x26=73728 data points, calculating the mean and stdev, then subtracting the mean and dividing by stdev for each of those points. Sorry to be so specific. But this is like a checksum to know I understand the concepts. $\endgroup$ – Alexander Soare Dec 27 '19 at 18:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.