I'm following this tutorial, and I wonder why is there a train-step - why is it necessary? I thought the whole idea of GPT-2 is that you do not need to train it on specific text domain, as it's already pre-trained on a large amount of data.
I've come to learn about GPT-2 through Robert Miles AI safety Youtube channel and intend to look into it in more detail.
From my current understanding, GPT-2 is pre-trained to "understand" "natural" language (for any definition of the words in quotes). However you would want it to not only understand general text but generate text similar to some specific "genre", e.g. scientific articles, youtube comments, twitter messages, you name it.
So using its pre-trained understanding, it analyzes the structure of sample texts and replicates this structure.
For scientific articles this structure could be:
- Context of research topic
- Introduction of researchers
- Explanation of methods/experiments/discoveries
- Results and interpretation
- Future research and application
For Youtube comments the structure is probably more chaotic but could include a vague reference to former comments, insults, nonsensical bar-grade philosophy, internet slang and smileys.
TL;DR: The domain specific text is only used to tell GPT-2 what you're looking for. You basically hand it context to work with, instead of prompting "Say something clever" (my least favorite line at parties, when I've been introduced as clever).
P.S.: Take this with a grain of salt. It's 90% conjecture from incomplete information.