# In Batch Normalisation, are $\hat{\mu}$, $\hat{\sigma}$ the mean and stdev of the original mini-batch or of the input into the current layer?

In Batch Normalisation, are the sample mean and standard deviation we normalise by the mean/sd of the original data put into the network, or of the inputs in the layer we are currently BN'ing over?

For instance, suppose I have a mini-batch size of 2 which contains $$\textbf{x}_1, \textbf{x}_2$$. Suppose now we are at the $$k$$th layer and the outputs from the previous layer are $$\tilde{\textbf{x}}_1,\tilde{\textbf{x}}_2$$. When we perform batch norm at this layer would be subtract the sample mean of $$\textbf{x}_1, \textbf{x}_2$$ or of $$\tilde{\textbf{x}}_1,\tilde{\textbf{x}}_2$$?

My intuition tells me that it must be the mean,sd of $$\tilde{\textbf{x}}_1,\tilde{\textbf{x}}_2$$ otherwise I don't think it would be normalised to have 0 mean and sd of 1.

So, if this layer receives an input $$\mathrm{x}=\left(x^{(1)} \ldots x^{(d)}\right)$$, the formula for normalizing the $$k^{th}$$ dimension of $$\mathrm{x}$$ would look as follows: $$\widehat{x}^{(k)}=\frac{x^{(k)}-\mathrm{E}\left[x^{(k)}\right]}{\sqrt{\operatorname{Var}\left[x^{(k)}\right]}}$$
Note that in practice a constant $$\epsilon$$ is also added under the square root in the denominator to ensure stability.
• Right. There are a few locations in the text where they define it in an explicit manner (like in the abstract or when they define $x$ ). In some places, they may also refer to the process as “normalizing the activations”. – user5093249 Jun 8 '20 at 17:26