How does a batch normalization layer work?

I understood that we normalize to input features in order to bring them on the same scale so that weights won't be learned in arbitrary fashion and training would be faster.

Then I studied about batch-normalization and observed that we can do the normalization for outputs of the hidden layers in following way:

Step 1: normalize the output of the hidden layer in order to have zero mean and unit variance a.k.a. standard normal (i.e. subtract by mean and divide by std dev of that minibatch).

Step 2: rescale this normalized vector to a new vector with new distribution having $$\beta$$ mean and $$\gamma$$ standard deviation, where both $$\beta$$ and $$\gamma$$ are trainable.

I did not understand the purpose of the second step. Why can't we just do the first step, make the vector standard normal, and then move forward? Why do we need to rescale the input of each hidden neuron to an arbitrary distribution which is learned (through beta and gamma parameters)?

Definition and Explaination

For how Batch Normalization works exactly, I'll suggest you to read the following papers:

The recent interpretation on How BN works is that it can reduce the high-order effect as mentioned in Ian Goodfellow's lecture. So it's not really about reducing the internal covariate shift.

Intuition

For how it works intuitively, you can think that we want to normalize the intermediate outputs (zero mean and unit variance) if the normalization won't remove too much useful information.

However, normalization may not be suitable for all intermediate outputs. So $$\beta$$ and $$\gamma$$ is introduced to provide additional flexibility, if normalization removes too much useful information then $$\beta$$ and $$\gamma$$ will learn to become the original mean and variance, making the BN layer an identity transformation, as if it doesn't exist.

In practice, $$\beta$$ and $$\gamma$$ won't become the original mean and variance, since all intermediate outputs can be normalized in some certain way without losing too much useful information. So you can think of it to be a customized normalization for each BN layer.

tl;dr

BN layer normalize the intermediate outputs in default, however, if the neural network find out that these intermediate outputs should not be normalized, then the neural network undos or provide more flexibility to the normalization.

• What is the "high-order effect"?
– nbro
Dec 24, 2019 at 21:39
• @nbro You can see Ian Goodfellow's lecture or this post. It's mainly about the high-order interactions. Dec 26, 2019 at 8:46
• It's funny because the post doesn't actually explain how high order terms interact with normalized vs not. Only saying that it does. Sep 19, 2020 at 14:49

Look, this is currently a quite contentious issue. J3soons answer already links to the original paper and another one. However, most of the evidence for batch normalization is still empirical, and there is very little theoretical explanation for why it works.

There are a number of competing theories out there. Also, there are a number of papers providing explanations for how batch normalization works in certain settings. However, there is no general theorem/explanation that can explain why the technique works.

Here's one paper analyzing how it works in simplified settings. This one provides another analysis of the batch norm (you can find their appendices on Arxiv). Here's another one which is the most recent peer reviewed one I could find. To quote this last paper:

Perhaps at the core of the confusion is that BatchNorm has many effects. It has been correlated to reducing covariate shift (Ioffe and Szegedy, 2015), enabling higher learning rates (Bjorck et al., 2018), improving initialization (Zhang et al., 2019), and improving conditioning (Desjardins et al., 2015), to name a few. These entangled effects make it difficult to properly study the technique.

Andrej Karpathy did a video explaining initialization of neural networks.

To summarize, activations like tanh and relu cause too many vanishing gradients if the distribution of inputs has too larger standard deviation (or the mean negative for relu).

Putting batch norm (without β and γ) before your tanh will prevent it becoming saturated (i.e. prevent vanishing gradients) as less input values will be on the flat tails (i.e. $$<-3$$ or $$>3$$).

However, adding in β and γ will allow the neural network to control the saturation optimally of the tanh.

Similarity, zero mean input to relu will mean half of the inputs are saturated (and have vanishing gradient). A positive β parameter will stop this from happening.

So you can think of the step 2 as optimizing the saturation of the activation function.

Of course you can put the batch norm after the activation function, and perhaps the second step wouldn't be as needed here.

• I thought it is weird to put BN between the model weights and its activation. IMO the order should be BN -> weights -> activation. so beta and gamma are absorbed to the layer's weights and don't show up as separate parameters to be optimized. I'm sure this has caused lots of discussion here and elsewhere. Oct 27, 2022 at 12:56
• @NikoNyrh yes I probably should have said beta and gamma are completely useless before a linear layer. Although I'm not sure about conv layers Oct 27, 2022 at 21:02