5 votes
Accepted

What is the difference between one-shot learning, transfer learning and fine tuning?

They are all related terms. From top to bottom: One-shot learning aims to achieve results with one or very few examples. Imagine an image classification task. You may show an apple and a knife to a ...
user avatar
  • 206
4 votes
Accepted

Is it possible that the fine-tuned pre-trained model performs worse than the original pre-trained model?

Yes, this is quite the expected behavior. The main difference between expected and current behavior lies in the amount of data you are using for training VS the amount of data that the pre-trained ...
user avatar
  • 1,038
1 vote

GPT-2: (Hardware) requirements for fine-tuning the 774M model

Possibly a bit late to the answer, but I doubt you'd be able to run GPT-2 774M in FP32 on 2070 Super which has 8GB VRAM. I know it's not an exact comparison, but fine-tuning BERT Large (345M) in FP32 ...
user avatar
1 vote
Accepted

When doing transfer learning, which initial layers do we need to freeze, and how should I change the last layer for my task?

After a lot of browsing online for answers to these questions, this is what I came up with. What are the initial layers in this case? How exactly can I freeze them? The initial few layers are said ...
user avatar
  • 143
1 vote
Accepted

Does BERT freeze the entire model body when it does fine-tuning?

Taken directly from HuggingFace Note that if you are used to freezing the body of your pretrained model (like in computer vision) the above may seem a bit strange, as we are directly fine-tuning the ...
user avatar
  • 21
1 vote
Accepted

Would this count as a Transfer Learning approach?

This tells me that because the initial model was performing so poorly without any fine-tuning, most of the learning that led to the 90% accuracy was only because of the additional layers, not the ...
user avatar
  • 24.5k
1 vote

What is the difference between feature extraction and fine-tuning in transfer learning?

The difference between the two approaches (feature extraction vs fine-tuning) is well explained here: Fine Tuning vs Joint Training vs Feature Extraction Also, this paper evaluate the performance one ...
user avatar
1 vote

How to fine tune BERT for question answering?

The answer is yes but 'lightweight' will require a 'lightweight' model. Your application for 'domain one' is called open domain question answering (ODQA). Here is a demonstration of ODQA using BERT: ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible