During transfer learning in computer vision, I've seen that the layers of the base model are frozen if the images aren't too different from the model on which the base model is trained on.

However, on the NLP side, I see that the layers of the BERT model aren't ever frozen. What is the reason for this?

  • $\begingroup$ Maybe you should provide the link to 2-3 examples (of implementations or papers/models) where the layers are not frozen (just to give more context). In any case, freezing the layers may not be necessarily required if you use a very low learning rate (but I am not an expert on this topic). $\endgroup$
    – nbro
    Commented Oct 3, 2020 at 14:38
  • $\begingroup$ I wouldn't say they are never frozen, just rarely. Generally, you fine-tune the entire model if you care more about accuracy (or whatever metric) while you freeze layers and fine-tune only the later layers if you care more about fine-tuning/prediction speed and memory usage. For example, fine-tuning the entire BERT on a GPU can take about 8GB of VRAM in my experience, which can be reduced by a gb or two by freezing layers. $\endgroup$ Commented Oct 5, 2020 at 17:20
  • $\begingroup$ @primussucks if we don't freeze bert new layers will mess up trained weights. isn't it so? $\endgroup$ Commented May 16, 2021 at 14:01
  • 2
    $\begingroup$ @datonefaridze It will change the weights yes, but "mess up" suggests that the model will perform poorly if they aren't frozen, which just isn't true. $\endgroup$ Commented May 18, 2021 at 7:53

1 Answer 1


Corrections and other answers are welcome, but here are a few thoughts:

There are several approaches in terms of which weights get frozen (and also other considerations, see for example Fig. 5 in "Galactica: A Large Language Model for Science").

Which of the approaches yields higher-quality results depends on the architecture (and hyperparameters) and dataset.

There can be rules of thumb, for example this old snapshot of a "Documentation" of Transformer architectures at Hugging Face said:

we are directly fine-tuning the whole model without taking any precaution. It actually works better this way for Transformers model

but this explanation apparently was removed from the new vesion of this page. Maybe it turned out that such rules of thumb aren't right in general.

Quality of results is also not the only thing being optimized. Some choices are made due to memory or compute considerations. For example, when freezing the first layers, their output features can be computed only once for all samples, saved, and used thereafter; moreover, computing the gradient of the loss with respect to the weights of the first frozen network block is not necessary.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .