# What is the difference between feature extraction and fine-tuning in transfer learning?

I'm building a model for facial expression recognition, and I want to use transfer learning. From what I understand, there are different steps to do it. The first is the feature extraction and the second is fine-tuning. I want to understand more about these two stages, and the difference between them. Must we use them simultaneously in the same training?

• Hello. Welcome to AI SE. Could you please provide the link the resource where you were reading that claims that "feature extraction" and "fine-tuning" are two different approaches to transfer learning? Because I don't think that's the case. I think these are just 2 stages of transfer learning, and maybe this would be the start of an answer to your question in the title, but I could be wrong.
– nbro
Jun 7, 2021 at 22:39
• Hello. Yeah I just realized that I was totaly wrong. Yes there are 2 stages in transfer leraning but should we use them in the same training or there is a difference ? Thank you for your reply. Jun 7, 2021 at 22:53

The difference between the two approaches (feature extraction vs fine-tuning) is well explained here: Fine Tuning vs Joint Training vs Feature Extraction

Also, this paper evaluate the performance one can hope to achieve with 2 sequence models (ELMo and BERT) with each approach: To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks

• Hello. Welcome to our community! Could you please provide more details and not just provide a link to an external source? We usually expect answers to be self-contained and links should only be used to provide additional details, unless the person was specifically asking for a reference/link.
– nbro
Jan 12 at 11:47
– Community Bot
Jan 12 at 17:33

Typically, in transfer learning, you have 2-3 stages

1. Pre-training: pre-train some base model $$M_\text{base}$$ on some "general" dataset $$A$$; note that you may not necessarily need to train $$M_\text{base}$$, but it may already be available e.g. on the web. During this phase, we extract (general) features or learn representations of the data, which can "bootstrap" the learning task with your specific dataset

2. Training: You replace the last layers of $$M_\text{base}$$ (i.e. the classifier/regression part) with new layers to solve your task, then you might freeze the initial layers (e.g. the convolutional layers) that are assumed to contain the general extracted features that can also be useful for your task: let's call this model $$M_\text{main}$$; at this point, you train this partially frozen model $$M_\text{main}$$ with your dataset $$B$$.

3. Fine-tuning: after training, you could unfreeze some of the frozen layers in $$M_\text{main}$$, especially the ones closest to your new classifier, then train again

In all 3 stages, one could say that we're extracting features (because we're learning weights), but some people, I guess, will refer to the pre-training phase as the feature extraction phase. I think I've seen people call the training stage also the fine-tuning stage (and the previous version of this answer actually was referring to the training phase as the fine-tuning phase), but, in the end, these terms could be used inconsistently anyway, so the important thing is that you understand what's going on and keep context into account.