I have been reading this TensorFlow tutorial on transfer learning, where they unfroze the whole model and then they say:

When you unfreeze a model that contains BatchNormalization layers in order to do fine-tuning, you should keep the BatchNormalization layers in inference mode by passing training=False when calling the base model. Otherwise the updates applied to the non-trainable weights will suddenly destroy what the model has learned.

My question is: why? The model's weights are adapting to the new data, so why do we keep the old mean and variance, which was calculated on ImageNet? This is very confusing.

  • 1
    $\begingroup$ Thanks for the clarification. When you ask a question next time, could you please keep in mind my edits to this post and format and present your post in a similar way? For example, you should always provide the link to the article that you're referring to, you should probably quote the parts of the article that you're confused about, etc. This would save me a little bit of time and I wouldn't need to ask for clarification so much. I hope you understand this. Thanks! $\endgroup$
    – nbro
    Dec 13 '20 at 13:35
  • $\begingroup$ @nbro okay next time i will take it into account $\endgroup$ Dec 13 '20 at 13:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.