Let us assume your dataset has $n$ training samples each of size $s$ and you divided them into $k$ batches for training. Then each batch has $n_k = \dfrac{n}{k}$ training samples.
Batch normalization can be applied to any input or hidden layer in a neural network. So, assume that I am applying batch normalization at every possible place I can.
Now, consider a particular batch normalization layer (say $b$) of a hidden layer $\ell$. Now, I am confused about the working frequency of $b$.
Will it be activated only after every $n_k - 1$ forward passes i.e, once per batch at the end of the batch? If no, then how $b$ calculates the mean and standard deviation for every forward pass while training if $n_k$ output vectors of $\ell$ are not available at that instant?
Will $b$ calculates the mean and standard deviated, for every forward pass, based on the outputs of $\ell$ that are calculated so far? If yes, then why it is called batch normalization?
To put it concisely, are batch normalization layers active for every iteration? If yes then how they are normalizing a "batch" of vectors?
You can check here which says
The mean and standard-deviation are calculated per-dimension over the mini-batches
