# What are the consequences of layer norm vs batch norm?

I'll start with my understanding of the literal difference between these two. First, let's say we have an input tensor to a layer, and that tensor has dimensionality $$B \times D$$, where $$B$$ is the size of the batch and $$D$$ is the dimensionality of the input corresponding to a single instance within the batch.

• Batch norm does the normalization across the batch dimension $$B$$
• Layer norm does the normalization across $$D$$

What are the differences in terms of the consequences of this choice?

This is how I understand it.

Batch normalization is used to remove internal covariate shift by normalizing the input for each hidden layer using the statistics across the entire mini-batch, which averages each individual sample, so the input for each layer is always in the same range. This can be seen from the BN equation:

$$\textrm{BN}(x)= \gamma\left(\frac{x-\mu(x)}{\sigma(x)}\right)+\beta$$

where $$\gamma$$ and $$\beta$$ are affine parameters learned from data; $$\mu(x)$$ and $$\sigma(x)$$ are the mean and standard deviation, computed across batch size and spatial dimensions independently for each feature channel. First, we normalize each channel with 0 mean and standard deviation of 1 according to batch statistics. We then scale and shift each channel with $$\gamma$$ and $$\beta$$.

This is fine if you want to classify an average object on an image from different viewing angles and lighting conditions. It is defined similarly to BN:

$$\textrm{LN}(x)= \gamma\left(\frac{x-\mu(x)}{\sigma(x)}\right)+\beta$$

but now $$\mu(x)$$ and $$\sigma(x)$$ are computed across all channels for each individual sample. Here's an illustration of the difference:

So layer normalization averages input across channels (for 2d input), which preserves the statistics of an individual sample. In some cases, we want to penalize the weights norm with respect to an individual sample rather than to the entire batch, as was done in WGAN-GP.

In terms of style transfer for images, it is also important to preserve the individual color statistics of a sample. Therefore, StyleGAN uses adaptive instance normalization, which is an extension of the original instance normalization, where each channel is normalized individually.

In addition, BN has several problems: the batch size must be large enough to capture overall statistics, which is sometimes impossible if you are working with large images since the model won't fit in memory. The concept of a batch is not always present, or it may change from time to time.

I strongly encourage you to read the original BN paper and also:
Group Normalization

• Thanks for your thoughts Aray. I'm just not sure about some of the things you say. For instance, I don't think batch norm "averages each individual sample". I also don't think layer norm "averages input across channels". It's very possible though, that what you mean to say is correct. I think my two key takeaways from your response are 1) Layer normalization might be useful if you want to maintain the distribution of pixels (or whatever constitutes a sample), and 2) batch norm might not make sense with small batch sizes. Apr 27, 2021 at 12:30
• I updated my answer by adding equations and illustrations to support my assertions. As I am also looking for better intuition and use cases, I look forward to any feedback and additions. Apr 27, 2021 at 13:33
• "Batch normalization is used to remove internal covariate shift". There's some controversy about it. ai.stackexchange.com/questions/27260/… Apr 27, 2021 at 18:03
• This is the original statement from the paper. The paper itself is titled "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift" Apr 27, 2021 at 18:30
• @Alex you can also call them scaling and offset parameters. The name is not important. What's important is that they are learnable and that they rescale and shift the sample statistics. Dec 2, 2021 at 8:54