# What are the consequences of layer norm vs batch norm?

I'll start with my understanding of the literal difference between these two. First, let's say we have an input tensor to a layer, and that tensor has dimensionality $$B \times D$$, where $$B$$ is the size of the batch and $$D$$ is the dimensionality of the input corresponding to a single instance within the batch.

• Batch norm does the normalization across the batch dimension $$B$$
• Layer norm does the normalization across $$D$$

What are the differences in terms of the consequences of this choice?

This is how I understand it.

Batch normalization is used to remove internal covariate shift by normalizing the input for each hidden layer using the statistics across the entire mini-batch, which averages each individual sample, so the input for each layer is always in the same range. This can be seen from the BN equation:

$$\textrm{BN}(x)= \gamma\left(\frac{x-\mu(x)}{\sigma(x)}\right)+\beta$$

where $$\gamma$$ and $$\beta$$ are affine parameters learned from data; $$\mu(x)$$ and $$\sigma(x)$$ are the mean and standard deviation, computed across batch size and spatial dimensions independently for each feature channel. First, we normalize each channel with 0 mean and standard deviation of 1 according to batch statistics. We then scale and shift each channel with $$\gamma$$ and $$\beta$$.

This is fine if you want to classify an average object on an image from different viewing angles and lighting conditions. It is defined similarly to BN:

$$\textrm{LN}(x)= \gamma\left(\frac{x-\mu(x)}{\sigma(x)}\right)+\beta$$

but now $$\mu(x)$$ and $$\sigma(x)$$ are computed across all channels for each individual sample. Here's an illustration of the difference: So layer normalization averages input across channels (for 2d input), which preserves the statistics of an individual sample. In some cases, we want to penalize the weights norm with respect to an individual sample rather than to the entire batch, as was done in WGAN-GP.

In terms of style transfer for images, it is also important to preserve the individual color statistics of a sample. Therefore, StyleGAN uses adaptive instance normalization, which is an extension of the original instance normalization, where each channel is normalized individually.

In addition, BN has several problems: the batch size must be large enough to capture overall statistics, which is sometimes impossible if you are working with large images since the model won't fit in memory. The concept of a batch is not always present, or it may change from time to time.

I strongly encourage you to read the original BN paper and also: