I'm following this tutorial, and I wonder why is there a train-step - why is it necessary? I thought the whole idea of GPT-2 is that you do not need to train it on specific text domain, as it's already pre-trained on a large amount of data.


I've come to learn about GPT-2 through Robert Miles AI safety Youtube channel and intend to look into it in more detail.

From my current understanding, GPT-2 is pre-trained to "understand" "natural" language (for any definition of the words in quotes). However you would want it to not only understand general text but generate text similar to some specific "genre", e.g. scientific articles, youtube comments, twitter messages, you name it.

So using its pre-trained understanding, it analyzes the structure of sample texts and replicates this structure.
For scientific articles this structure could be:

  • Abstract
  • Context of research topic
  • Introduction of researchers
  • Explanation of methods/experiments/discoveries
  • Results and interpretation
  • Future research and application

For Youtube comments the structure is probably more chaotic but could include a vague reference to former comments, insults, nonsensical bar-grade philosophy, internet slang and smileys.

TL;DR: The domain specific text is only used to tell GPT-2 what you're looking for. You basically hand it context to work with, instead of prompting "Say something clever" (my least favorite line at parties, when I've been introduced as clever).

P.S.: Take this with a grain of salt. It's 90% conjecture from incomplete information.

  • $\begingroup$ your not far off, but thats only a piece of it, it also extends to task specificity in an overarching vague sense. In other words, it depends what you want to get out of it. $\endgroup$ – mshlis Sep 3 at 13:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.