# Why does Batch Normalization work?

Adding BatchNorm layers improves training time and makes the whole deep model more stable. That's an experimental fact that is widely used in machine learning practice.

My question is - why does it work?

The original (2015) paper motivated the introduction of the layers by stating that these layers help fixing "internal covariate shift". The rough idea is that large shifts in the distributions of inputs of inner layers makes training less stable, leading to a decrease in the learning rate and slowing down of the training. Batch normalization mitigates this problem by standardizing the inputs of inner layers.

This explanation was harshly criticized by the next (2018) paper -- quoting the abstract:

... distributional stability of layer inputs has little to do with the success of BatchNorm

They demonstrate that BatchNorm only slightly affects the inner layer inputs distributions. More than that -- they tried to inject some non-zero mean/variance noise into the distributions. And they still got almost the same performance.

Their conclusion was that the real reason BatchNorm works was that...

Instead BatchNorm makes the optimization landscape significantly smoother.

Which, to my taste, is slightly tautological to saying that it improves stability.

I've found two more papers trying to tackle the question: In this paper the "key benefit" is claimed to be the fact that Batch Normalization biases residual blocks towards the identity function. And in this paper that it "avoids rank collapse".

So, is there any bottom line? Why does BatchNorm work?

• This blog post attempted to explain why this is the case (go to the part "So, why does Batch Norm Work?"), but as they mentioned it is kind of a speculative thought.
– Enes
Feb 12 at 11:43

I believe anything in machine learning that works, works because it flattens and smoothens the loss landscape.

Batch and layer normalization would help ensure that the feature vectors (i.e. channels) are embedded around the unit sphere Batch/Instance norm translates to origin. Layer norm scales radially to unit sphere. Viewing neural networks as transformations, this would make the loss landscape smoother since the transformations the neural net needs to find would be more "regular".

I would recomend this video to learn about loss landscapes.

This got me thinking about my understanding of batch normalization. I thought I understand it until I read this. Then, I refer to the Coursera deep learning specialization by Andrew Ng.
Prof. Andrew Ng explained it this way.

One reason why does batch norm work is that it normalizes not only the input features but also further values in the hidden units to take on a similar range of values that can speed up learning.

The second reason why batch norm works, is it makes weights, later or deeper than the network you have, say the weight on layer 10, more robust to changes to weights in earlier layers of the neural network (eg. in layer one). However, these hidden unit values are changing all the time, and so it's suffering from the problem of covariate shift. So what batch norm does, is it reduces the amount that the distribution of these hidden unit values shifts around. What batch norm ensures is that no matter how the parameters of the neural network update, their mean and variance will at least stay the same mean and variance, causing the input values to become more stable, so that the later layers of the neural network has more firm ground to stand on.

And even though the input distribution changes a bit, it changes less, and what this does is, even as the earlier layers keep learning, the amounts that this forces the later layers to adapt to as early as layer changes is reduced or, if you will, it weakens the coupling between what the early layers parameters has to do and what the later layers parameters have to do. And so it allows each layer of the network to learn by itself, a little bit more independently of other layers, and this has the effect of speeding up of learning in the whole network. Takeaway is that batch norm means that, especially from the perspective of one of the later layers of the neural network, the earlier layers don't get to shift around as much, because they're constrained to have the same mean and variance. And so this makes the job of learning on the later layers easier. It turns out batch norm has a second effect, it has a slight regularization effect. So one non-intuitive thing of a batch norm is that each mini-batch, the mean and variance computed on just that mini-batch as opposed to computed on the entire data set, that mean and variance has a little bit of noise in it, because it's computed just on your mini-batch of, say, 64, or 128, or maybe 256 or larger training examples. Batch norm works with mini-batch

• The punctuation needs some work. Feb 8 at 14:58
• @HappyFace, I see your point. Feb 9 at 4:47
• Hey, @kibromhft, my question specifically mentions these kinds of explanations. Experiments in the 2018 paper show that BatchNorm doesn't affect distributions in inner layers. Even more - artificially changing these distributions doesn't affect the performance much. Feb 9 at 10:03

When we are training deep neural Network gradient tells how to update each parameter, under the assumption other layers do not change.In Practice, we update all the layers simultaneously. When we update, unexpected results can happen because many functions composed together are changed simultaneously using updates that were computed under the assumption that other function remains constant.This makes it very hard to choose an appropriate leaning rate, because the effects of an update to the parametrs of one layer strongly on all other layers.

How does Batch Normalisation Help : Batch Normalisation a layer which is added to any input or hidden layer in the neural network. Suppose H is the minitach of activations of the layer to normalize.

The formula for normalizing H is :

$$_H = \frac{H - Mean}{Standard Deviation}$$

Mean : Vector Containing Mean of each unit Standard Deviation : Vector Containing Mean of each unit

At training time mean and sd are calculated and when we backpropogate through these operations for apply mean, sd and Normalize H. This means that gradient will never propose an operation that acts simply to increase the standard deviation and mean of hi, the normalization operation remove the effect of such an action and zero out its componenr in the gradient. Hence Batch Normalisation thus ensure no or slight covariance shift in the input to layer after Batch Normalisation and thus improving learning time as shown in the original paper mentioned in question.

For more details : https://www.deeplearningbook.org/contents/optimization.html

• At least this "tells how to update each parameter, under the assumption other layers do not change.In Practice, we update all the layers simultaneously." seems to be copied from here. Either edit this answer to clarify which parts you copied or this answer will be deleted.
– nbro
Jan 6 at 11:39
• if you look at the reference we are referring to same reference. Nothing has been copied from the link :) The sentence you have highlighted is taken from reference. Jan 6 at 11:44
• Yes, I understood that the reference is the same, but you cannot just copy from a reference without explicitly showing us WHAT you copied, i.e. you need to quote. To quote, you need to precede the text that you copied with >. Take a look at ai.stackexchange.com/help/referencing to know more about how to quote parts of other answers. Please, take the time to go through all your answers to quote things that you copied (from other sources) in order for them not to be deleted.
– nbro
Jan 6 at 11:46
• Thanks i am new to this platform , will keep this things in mind Jan 6 at 11:50
• Don't just put the reference at the end. Whenever you quote something, tell us immediately where you copied it from, which section and page, so that people can check that you're referencing a specific part of another source. That chapter 8 is a bit long.
– nbro
Jan 6 at 11:53

To some extend, it get rid of low intensity numerical noise. Condition properties of the optimization problem is always an issue, i suspect BatchNorm alleviate this instability.

• Adding some artificial noise to neural networks quite often increases their generalization performance. Apr 16, 2021 at 18:35
• I was just thinking about the conditioning that BatchNorm performs and that also in some cases this process cutoff low intensity component. When a feature appears on a layer, it become evident since its intensity is very over average. When you "normalize", all others components falls under threshold at least numerically. Then this "kind of noise" is removed. Another thing is adding uniform noise, which helps to search such a desirable feature and it is not removed by "normalization". Apr 20, 2021 at 14:38

It is a question with no simple answer.

On one hand the BatchNormalization is unloved by some arguing it doesn't change the accuracy of neural networks or biased them. On the other hand, it is highly recommended by the other because it leads to better trained models with a larger scope of predictions and less chances of overflow.

All I know for sure is that BN is really efficient on image classification. In fact, like the image categorization and classification soar this last years and that BN is a good practice in this field, it has spread to almost all DNNs.

Not only is the BN not always used in the right purpose, but it is often used without taking into account several elements such as :

• The layers between which apply BN
• The initializer algorithms
• The activation algorithms
• etc

For more computer sciences litterature "against" BN, I will let you look at the H. Zhang et al paper who has trained a DNN without BN and get good results.

Some people use Gradient Clipping technique (R. Pascanu) instead of the BN in particular for RNNs

I hope it will give you some answers !

• Thanks for the response. I'm following it as I added the bounty. I down voted it but don't take offence! The reason I did it is because I don't believe any part of it really addresses the question. Votes help others quickly identify the most relevant responses. Jun 28, 2021 at 21:29
• Thanks for the feedback and the explanation. So I guess I didn't understand the question. Can you just reformulate it to let me understand it please ? Jun 29, 2021 at 16:40
• I think your answer is more for a question like "Where is BN useful vs not?" or "Is BN always a good thing to use?". The question asks "What is the generally accepted reason for why BN works well?" Like if I ask "Why do conv kernels work well for images?" the answer is along the lines of: "Because it satisfies the inductive priors of translational invariance, feature hierarchies, and to some degree, scale invariance". Your answer doesn't talk about the properties of BN that make it successful. And in this case OP also wants the answer to be backed by authoritative sources Jun 30, 2021 at 8:34