1
$\begingroup$

When should one not use normalization between layers like Batch Norm, Layer Norm, Instance Norm, and Group Norm in deep learning while training a DL model?

$\endgroup$

1 Answer 1

0
$\begingroup$

In transfer learning cases when fine-tuning a pre-trained model on a new dataset, your mentioned various types of normalization layers might be frozen or omitted to avoid disrupting their already learned statistics (mean and standard deviation) that are well-suited for the original dataset. When fine-tuning a pre-trained model, there is a risk of "catastrophic forgetting" where the model forgets what it learned during the original training. If normalization layers are allowed to update their statistics during fine-tuning, it might disrupt the knowledge catastrophically captured by the pre-trained model.

Furthermore, if your fine-tuning target dataset is relatively small, freezing or omitting normalization layers can help prevent overfitting since they effectively introduce potentially many additional parameters particularly such as batch normalization's scale and shift of each unit that could contribute to overfitting.

In some generative models like GANs, normalization layers may interfere with the learning process. The generator in a GAN is trained through an adversarial process where it constantly adjusts its parameters based on the evolving dynamics of the discriminator. Various normalization meta-algorithms assume that the distribution of inputs remains constant within a batch which may not hold during GAN training, causing internal covariate shift ironically. GAN training is already known for its inherent instability challenges and the introduction of normalization layers can sometimes lead to further instability, making it harder for the model to converge.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .