Adding BatchNorm layers improves training time and makes the whole deep model more stable. That's an experimental fact that is widely used in machine learning practice.

My question is - why does it work?

The original (2015) paper motivated introduction of the layers by stating that these layers help fixing "internal covariate shift". The rough idea is that large shifts in the distributions of inputs of inner layers makes training less stable, leading to decrease in the learning rate and slowing down of the training. Batch normalization mitigates this problem by standardizing the inputs of inner layers.

This explanation was harshly criticized by the next (2018) paper -- quoting the abstact:

... distributional stability of layer inputs has little to do with the success of BatchNorm

They demonstrate that BatchNorm only slightly affects the inner layer inputs distributions. More than that -- they tried to inject some non-zero mean/variance noise into the distributions. And they still got almost the same performance.

Their conclusion was that the real reason BatchNorm works was that...

Instead BatchNorm makes the optimization landscape significantly smoother.

Which, to my taste, is slightly tautological to saying that it improves stability.

I've found two more papers trying to tackle the question: In this paper the "key benefit" is claimed to be the fact that Batch Normalization biases residual blocks towards the identity function. And in this paper that it "avoids rank collapse".

So. Is there any bottom line? Why does BatchNorm work?

  • 4
    $\begingroup$ This is an amazing question. I thought it was going to be an introductory one, but you clearly did your research, exposed it clearly and made a very profound one. Bravo! $\endgroup$
    – Alpha
    Apr 15 at 2:08

To some extend, it get rid of low intensity numerical noise. Condition properties of the optimization problem is always an issue, i suspect BatchNorm alleviate this instability.

  • 1
    $\begingroup$ Adding some artificial noise to neural networks quite often increases their generalization performance. $\endgroup$
    – Kostya
    Apr 16 at 18:35
  • $\begingroup$ I was just thinking about the conditioning that BatchNorm performs and that also in some cases this process cutoff low intensity component. When a feature appears on a layer, it become evident since its intensity is very over average. When you "normalize", all others components falls under threshold at least numerically. Then this "kind of noise" is removed. Another thing is adding uniform noise, which helps to search such a desirable feature and it is not removed by "normalization". $\endgroup$ Apr 20 at 14:38

It is a question with no simple answer.

On one hand the BatchNormalization is unloved by some arguing it doesn't change the accuracy of neural networks or biased them. On the other hand, it is highly recommended by the other because it leads to better trained models with a larger scope of predictions and less chances of overflow.

All I know for sure is that BN is really efficient on image classification. In fact, like the image categorization and classification soar this last years and that BN is a good practice in this field, it has spread to almost all DNNs.

Not only is the BN not always used in the right purpose, but it is often used without taking into account several elements such as :

  • The layers between which apply BN
  • The initializer algorithms
  • The activation algorithms
  • etc

For more computer sciences litterature "against" BN, I will let you look at the H. Zhang et al paper who has trained a DNN without BN and get good results.

Some people use Gradient Clipping technique (R. Pascanu) instead of the BN in particular for RNNs

I hope it will give you some answers !

  • 3
    $\begingroup$ Thanks for the response. I'm following it as I added the bounty. I down voted it but don't take offence! The reason I did it is because I don't believe any part of it really addresses the question. Votes help others quickly identify the most relevant responses. $\endgroup$ Jun 28 at 21:29
  • $\begingroup$ Thanks for the feedback and the explanation. So I guess I didn't understand the question. Can you just reformulate it to let me understand it please ? $\endgroup$
    – himiro
    Jun 29 at 16:40
  • $\begingroup$ I think your answer is more for a question like "Where is BN useful vs not?" or "Is BN always a good thing to use?". The question asks "What is the generally accepted reason for why BN works well?" Like if I ask "Why do conv kernels work well for images?" the answer is along the lines of: "Because it satisfies the inductive priors of translational invariance, feature hierarchies, and to some degree, scale invariance". Your answer doesn't talk about the properties of BN that make it successful. And in this case OP also wants the answer to be backed by authoritative sources $\endgroup$ Jun 30 at 8:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.