Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
I'll start with my understanding of the literal difference between these two. First, let's say we have an input tensor to a layer, and that tensor has dimensionality $B \times D$, where $B$ is the size of the batch and $D$ is the dimensionality of the input corresponding to a single instance within the batch.
What are the differences in terms of the consequences of this choice?
This is how I understand it.
Batch normalization is used to remove internal "covariate shift" (wich may be not the case) by normalizing the input for each hidden layer using the statistics across the entire mini-batch, which averages each individual sample, so the input for each layer is always in the same range. This can be seen from the BN equation:
$$ \textrm{BN}(x)= \gamma\left(\frac{x-\mu(x)}{\sigma(x)}\right)+\beta $$
where $\gamma$ and $\beta$ are affine parameters learned from data; $\mu(x)$ and $\sigma(x)$ are the mean and standard deviation, computed across batch size and spatial dimensions independently for each feature channel. First, we normalize each channel with 0 mean and standard deviation of 1 according to batch statistics. We then scale and shift each channel with $\gamma$ and $\beta$.
This is fine if you want to classify an average object on an image from different viewing angles and lighting conditions. It is defined similarly to BN:
$$ \textrm{LN}(x)= \gamma\left(\frac{x-\mu(x)}{\sigma(x)}\right)+\beta $$
but now $\mu(x)$ and $\sigma(x)$ are computed across all channels for each individual sample. Here's an illustration of the difference:
So layer normalization averages input across channels (for 2d input), which preserves the statistics of an individual sample. In some cases, we want to penalize the weights norm with respect to an individual sample rather than to the entire batch, as was done in WGAN-GP.
In terms of style transfer for images, it is also important to preserve the individual color statistics of a sample. Therefore, StyleGAN uses adaptive instance normalization, which is an extension of the original instance normalization, where each channel is normalized individually.
In addition, BN has several problems: the batch size must be large enough to capture overall statistics, which is sometimes impossible if you are working with large images since the model won't fit in memory. The concept of a batch is not always present, or it may change from time to time.
I strongly encourage you to read the original BN paper and also:
Adaptive instance normalization
Group Normalization
