I'm building a model for facial expression recognition, and I want to use transfer learning. From what I understand, there are different steps to do it. The first is the feature extraction and the second is fine-tuning. I want to understand more about these two stages, and the difference between them. Must we use them simultaneously in the same training?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ Hello. Welcome to AI SE. Could you please provide the link the resource where you were reading that claims that "feature extraction" and "fine-tuning" are two different approaches to transfer learning? Because I don't think that's the case. I think these are just 2 stages of transfer learning, and maybe this would be the start of an answer to your question in the title, but I could be wrong. $\endgroup$– nbroJun 7, 2021 at 22:39
-
$\begingroup$ Hello. Yeah I just realized that I was totaly wrong. Yes there are 2 stages in transfer leraning but should we use them in the same training or there is a difference ? Thank you for your reply. $\endgroup$– SpeedskillsxJun 7, 2021 at 22:53
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
2
The difference between the two approaches (feature extraction vs fine-tuning) is well explained here: Fine Tuning vs Joint Training vs Feature Extraction
Also, this paper evaluate the performance one can hope to achieve with 2 sequence models (ELMo and BERT) with each approach: To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks
-
$\begingroup$ Hello. Welcome to our community! Could you please provide more details and not just provide a link to an external source? We usually expect answers to be self-contained and links should only be used to provide additional details, unless the person was specifically asking for a reference/link. $\endgroup$– nbroJan 12, 2022 at 11:47
-
1$\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$– Community BotJan 12, 2022 at 17:33
$\begingroup$
$\endgroup$
4
Typically, in transfer learning, you have 2-3 stages
Pre-training: pre-train some base model $M_\text{base}$ on some "general" dataset $A$; note that you may not necessarily need to train $M_\text{base}$, but it may already be available e.g. on the web. During this phase, we extract (general) features or learn representations of the data, which can "bootstrap" the learning task with your specific dataset
Training: You replace the last layers of $M_\text{base}$ (i.e. the classifier/regression part) with new layers to solve your task, then you might freeze the initial layers (e.g. the convolutional layers) that are assumed to contain the general extracted features that can also be useful for your task: let's call this model $M_\text{main}$; at this point, you train this partially frozen model $M_\text{main}$ with your dataset $B$.
Fine-tuning: after training, you could unfreeze some of the frozen layers in $M_\text{main}$, especially the ones closest to your new classifier, then train again
In all 3 stages, one could say that we're extracting features (because we're learning weights), but some people, I guess, will refer to the pre-training phase as the feature extraction phase. I think I've seen people call the training stage also the fine-tuning stage (and the previous version of this answer actually was referring to the training phase as the fine-tuning phase), but, in the end, these terms could be used inconsistently anyway, so the important thing is that you understand what's going on and keep context into account.
You can find more information about this topic here. Note that there may be other more sophisticated or simply different approaches to transfer learning.
-
1$\begingroup$ Fine-tuning most often does not include freezing the previous trained layers. Actually it is more likely related to training the whole model for a few more iterations on the task specific dataset after pretraining on the task-unrelated dataset. $\endgroup$ Jan 12, 2022 at 14:42
-
$\begingroup$ @Marcel_marcel1991 Well, during fine-tuning, you don't need to unfreeze all layers, but maybe you're right that some people may use fine-tuning to mean specifically when you unfreeze some of the layers. The TF tutorial says, for instance, "Also, you should try to fine-tune a small number of top layers rather than the whole MobileNet model." I will edit my answer to point that out. $\endgroup$– nbroJan 12, 2022 at 14:56
-
$\begingroup$ @Marcel_marcel1991 If you look at the answer on Stats that the other answer links to, they are using the word fine-tuning to refer to something slightly different (i.e. learning new tasks or classes continuously by slightly changing the output layer), and it's in the context of continual learning, rather than transfer learning (as described in TF tutorial and in other papers). So, as I say now in my answer, it's possible people may be using these terms to refer to slightly different things. $\endgroup$– nbroJan 12, 2022 at 15:24
-
$\begingroup$ I agree with you that there is no clear definition. I was only relating to that your answer assumes that you would always freeze the previous layers in fine-tuning, which is not the case. The edit now is fine to me. $\endgroup$ Jan 12, 2022 at 15:27