I am reading a survey on various normalization techniques adopted in neural network architectures.
The purpose of introducing normalization is understandable - to stabilize the training and avoid covariate shifts.
There is a plethora of proposed approaches:
- Batch Normalization. Probably, the most well-known approach. One averages over the batch and spatial dimensions and gets the mean and std vectors of size
(num_channels,): $$ \mu_c = \sum_{n, h, w}^{N, H, W} x_{nchw} \quad \sigma_c = \sqrt{\frac{1}{NHW}\sum_{n, h, w}^{N, H, W}(x_{nchw} - \mu_c)^2} $$ - Layer Normalization. This technique became very popular after the success of Transformer architectures. The average is over the channel and spatial dimensions and gets the mean and std vectors of size
(batch_size, num_channels): $$ \mu_n = \sum_{c, h, w}^{C, H, W} x_{nchw} \quad \sigma_n = \sqrt{\frac{1}{NHW}\sum_{c, h, w}^{C, H, W}(x_{nchw} - \mu_n)^2} $$ - Instance Normalization. This approach is popular in style transfer applications. The average is over the channel and spatial dimensions and gets the mean and std vectors of size
(batch_size,): $$ \mu_{nc} = \sum_{n, h, w}^{N, H, W} x_{nchw} \quad \sigma_{nc} = \sqrt{\frac{1}{NHW}\sum_{n, h, w}^{N, H, W}(x_{nchw} - \mu_{nc})^2} $$ There are much more approaches, but I listed these 3 as the simplest.
Then there are trainable parameters $\gamma$ and $\beta$, and the final output is: $$ \gamma \left(\frac{x - \mu(x)}{\sigma(x)}\right) + \beta $$
As far as I understand batch normalization forces weights to output something like $\mathcal{N}(\beta, \gamma)$ (normal distribution with mean $\beta$ and std $\gamma$). However, there are problems when batch size is small since the estimate would be inaccurate. Also, it seems to average over all images in the batch, but if there are different classes, probably one would like to have them to be distributed slightly different. This choice is the most widely used in CNN still, despite some [recent work] (https://arxiv.org/abs/2102.06171) says, that this layer can be replaced by another strategy.
Layer normalization seems to equalize different channels. Is there some intuition why it has to be so? Why do we need to make output activations similar to each other?
Instance normalization seems to be the most specific in the list. But I have not seen a lot of usage of this outside style transfer and GAN's.
Overall, the ultimate question is - how to choose a particular normalization strategy for the given problem and architecture?
