Let us assume your dataset has $n$ training samples each of size $s$ and you divided them into $k$ batches for training. Then each batch has $n_k = \dfrac{n}{k}$ training samples.

Batch normalization can be applied to any input or hidden layer in a neural network. So, assume that I am applying batch normalization at every possible place I can.

Now, consider a particular batch normalization layer (say $b$) of a hidden layer $\ell$. Now, I am confused about the working frequency of $b$.

Will it be activated only after every $n_k - 1$ forward passes i.e, once per batch at the end of the batch? If no, then how $b$ calculates the mean and standard deviation for every forward pass while training if $n_k$ output vectors of $\ell$ are not available at that instant?

Will $b$ calculates the mean and standard deviated, for every forward pass, based on the outputs of $\ell$ that are calculated so far? If yes, then why it is called batch normalization?

To put it concisely, are batch normalization layers active for every iteration? If yes then how they are normalizing a "batch" of vectors?

You can check here which says

The mean and standard-deviation are calculated per-dimension over the mini-batches

  • 1
    $\begingroup$ If I understand your question correctly, I think this is related to the momentum which calculates the moving average, hence it can be applied at every iteration. More Details. $\endgroup$
    – Yahya
    Jul 31, 2021 at 19:30
  • $\begingroup$ @Yahya Does the word "batch" here also need to be interpreted as the "collection" rather than collection of $\dfrac{n}{k}$ samples? $\endgroup$
    – hanugm
    Jul 31, 2021 at 23:39
  • $\begingroup$ @Yahya you can see that "the layer normalizes its output using the mean and standard deviation of the current batch of inputs" this statement says that we use mean and standard deviation on a batch. Even we use moving averages, the number of samples we are considering for normalization depends on number of iterations occurred till now. So, batch in batch normalization also refers to the collection of vectors we encountered till now. $\endgroup$
    – hanugm
    Jul 31, 2021 at 23:45
  • $\begingroup$ And it is also useful to check here which says The mean and standard-deviation are calculated per-dimension over the mini-batches. The word mini-batches here imposes the requirement that at-least mini batch number of vectors should be available to proceed. Am i wrong? $\endgroup$
    – hanugm
    Jul 31, 2021 at 23:47
  • $\begingroup$ @Yahya I am opining that the words batch and mini-batch are used loosely If they are considering moving averages at every forward pass. $\endgroup$
    – hanugm
    Jul 31, 2021 at 23:49


You must log in to answer this question.

Browse other questions tagged .