I have been reading this TensorFlow tutorial on transfer learning, where they unfroze the whole model and then they say:
When you unfreeze a model that contains
BatchNormalizationlayers in order to do fine-tuning, you should keep the
BatchNormalizationlayers in inference mode by passing
training=Falsewhen calling the base model. Otherwise the updates applied to the non-trainable weights will suddenly destroy what the model has learned.
My question is: why? The model's weights are adapting to the new data, so why do we keep the old mean and variance, which was calculated on ImageNet? This is very confusing.