10
$\begingroup$

In the official blog post about ChatGPT from OpenAI, there is this paragraph explaining how ChatGPT model was trained:

We trained this model using Reinforcement Learning from Human Feedback (RLHF), using the same methods as InstructGPT, but with slight differences in the data collection setup. We trained an initial model using supervised fine-tuning: human AI trainers provided conversations in which they played both sides—the user and an AI assistant. We gave the trainers access to model-written suggestions to help them compose their responses. We mixed this new dialogue dataset with the InstructGPT dataset, which we transformed into a dialogue format.

Especially this part:

We trained an initial model using supervised fine-tuning

My question is about the said initial model, is it some new model that has been trained from scratch or is it a GPT-3 model that has been fine tuned for specific tasks resulting in GPT-3.5 series ?

On the other hand, from the InstructGPT blog post, it is clearly stated that:

To make our models safer, more helpful, and more aligned, we use an existing technique called reinforcement learning from human feedback (RLHF). On prompts submitted by our customers to the API,our labelers provide demonstrations of the desired model behavior, and rank several outputs from our models. We then use this data to fine-tune GPT-3.

So does this mean that GPT-3.5 series models (and consequently ChatGPT) are fine-tuned from GPT-3 base model ?

$\endgroup$

2 Answers 2

13
$\begingroup$

I'll complement nbro's answer with this great visual summary by Yao Fu <[email protected]>:

enter image description here

$\endgroup$
1
  • 2
    $\begingroup$ This is exaclty what I was looking for. Thanks ! $\endgroup$
    – iMad
    Feb 13, 2023 at 8:19
7
$\begingroup$

ChatGPT has not been trained from scratch. ChatGPT is a fine-tuned version of a model from the GPT-3.5 series. OpenAI writes

ChatGPT is fine-tuned from a model in the GPT-3.5 series, which finished training in early 2022. You can learn more about the 3.5 series here.

Which models are in the GPT-3.5 series? You can read more about that in the linked blog post.

GPT-3.5 series is a series of models that was trained on a blend of text and code from before Q4 2021. The following models are in the GPT-3.5 series:

  • code-davinci-002 is a base model, so good for pure code-completion tasks
  • text-davinci-002 is an InstructGPT model based on code-davinci-002
  • text-davinci-003 is an improvement on text-davinci-002

So, ChatGPT must be a fine-tuned version of one of these 3 models, assuming the information in their site is accurate and up-to-date.

Now, according to this blog post

The Codex models are descendants of our GPT-3 models that can understand and generate code. Their training data contains both natural language and billions of lines of public code from GitHub

code-davinci-002 is a codex model. So, ChatGPT might be a descendent of GPT-3. I don't know what "descendent" exactly means here. Does it mean just fine-tuned or maybe a modified version?

Here they write that text-davinci-003 is the most capable GPT-3 model. Based on the information above, text-davinci-002 is an InstructGPT model based on code-davinci-002.

Here they write

We then use this data to fine-tune GPT-3.

The resulting InstructGPT models are much better at following instructions than GPT-3

So, InstructGPT models are fine-tuned GPT-3 models. That most likely implies that text-davinci-002 is a GPT-3 model and the only thing that changes is how it was trained. However, they also write

Our labelers prefer outputs from our 1.3B InstructGPT model over outputs from a 175B GPT-3 model, despite having more than 100x fewer parameters.

So, there isn't just one GPT3 model. However, the original GPT3 model had 175 billion parameters.

I still need to read the InstructGPT and GPT3 papers. Once I've done that, I may have more useful/concrete info, then I will update this answer.

$\endgroup$
2
  • $\begingroup$ Thanks for you answer. I went through more or less the same links (that I've included in my question) and I also checked the InstructGPT paper. The different statements from the blog posts are really confusing especially when it comes to different number of parameters... Maybe they use the last layers from GPT-3 to fine tune the new models with fewer parameters.. but in this case they need to run the input through GPT-3 everytime and I'm not sure this is a practical way to do... Anyway I'm waiting for your update. ^^ $\endgroup$
    – iMad
    Feb 4, 2023 at 17:52
  • 2
    $\begingroup$ I don't know when I will start reading them. I'll start a project at work that involves these OpenAI models, so I will most likely read these papers (at least to some extent), but I don't know when yet. $\endgroup$
    – nbro
    Feb 4, 2023 at 19:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .