2
$\begingroup$

In Batch Normalisation, are the sample mean and standard deviation we normalise by the mean/sd of the original data put into the network, or of the inputs in the layer we are currently BN'ing over?

For instance, suppose I have a mini-batch size of 2 which contains $\textbf{x}_1, \textbf{x}_2$. Suppose now we are at the $k$th layer and the outputs from the previous layer are $\tilde{\textbf{x}}_1,\tilde{\textbf{x}}_2$. When we perform batch norm at this layer would be subtract the sample mean of $\textbf{x}_1, \textbf{x}_2$ or of $\tilde{\textbf{x}}_1,\tilde{\textbf{x}}_2$?

My intuition tells me that it must be the mean,sd of $\tilde{\textbf{x}}_1,\tilde{\textbf{x}}_2$ otherwise I don't think it would be normalised to have 0 mean and sd of 1.

$\endgroup$
1
$\begingroup$

Your intuition is correct. We will be normalizing the inputs of the layer under consideration (just right before applying the activation function).

So, if this layer receives an input $\mathrm{x}=\left(x^{(1)} \ldots x^{(d)}\right)$, the formula for normalizing the $k^{th}$ dimension of $\mathrm{x}$ would look as follows: $$\widehat{x}^{(k)}=\frac{x^{(k)}-\mathrm{E}\left[x^{(k)}\right]}{\sqrt{\operatorname{Var}\left[x^{(k)}\right]}}$$

Note that in practice a constant $\epsilon$ is also added under the square root in the denominator to ensure stability.

Source: The original Batch Normalization paper (Section 3).

Andrew Ng's video on this topic might also be useful for illustration.

$\endgroup$
2
  • 1
    $\begingroup$ Thanks for clarifying this! it was actually the original paper I read that confused me. $\endgroup$ – David Ireland Jun 8 '20 at 12:45
  • 1
    $\begingroup$ Right. There are a few locations in the text where they define it in an explicit manner (like in the abstract or when they define $x$ ). In some places, they may also refer to the process as “normalizing the activations”. $\endgroup$ – user5093249 Jun 8 '20 at 17:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.