Are GPT-3.5 series models based on GPT-3?

Question

In the official blog post about ChatGPT from OpenAI, there is this paragraph explaining how ChatGPT model was trained:

We trained this model using Reinforcement Learning from Human Feedback (RLHF), using the same methods as InstructGPT, but with slight differences in the data collection setup. We trained an initial model using supervised fine-tuning: human AI trainers provided conversations in which they played both sides—the user and an AI assistant. We gave the trainers access to model-written suggestions to help them compose their responses. We mixed this new dialogue dataset with the InstructGPT dataset, which we transformed into a dialogue format.

Especially this part:

We trained an initial model using supervised fine-tuning

My question is about the said initial model, is it some new model that has been trained from scratch or is it a GPT-3 model that has been fine tuned for specific tasks resulting in GPT-3.5 series ?

On the other hand, from the InstructGPT blog post, it is clearly stated that:

To make our models safer, more helpful, and more aligned, we use an existing technique called reinforcement learning from human feedback (RLHF). On prompts submitted by our customers to the API,our labelers provide demonstrations of the desired model behavior, and rank several outputs from our models. We then use this data to fine-tune GPT-3.

So does this mean that GPT-3.5 series models (and consequently ChatGPT) are fine-tuned from GPT-3 base model ?

Answer 1

I'll complement nbro's answer with this great visual summary by Yao Fu <[email protected]>:

Answer 2

ChatGPT has not been trained from scratch. ChatGPT is a fine-tuned version of a model from the GPT-3.5 series. OpenAI writes

ChatGPT is fine-tuned from a model in the GPT-3.5 series, which finished training in early 2022. You can learn more about the 3.5 series here.

Which models are in the GPT-3.5 series? You can read more about that in the linked blog post.

GPT-3.5 series is a series of models that was trained on a blend of text and code from before Q4 2021. The following models are in the GPT-3.5 series:

• code-davinci-002 is a base model, so good for pure code-completion tasks
• text-davinci-002 is an InstructGPT model based on code-davinci-002
• text-davinci-003 is an improvement on text-davinci-002

So, ChatGPT must be a fine-tuned version of one of these 3 models, assuming the information in their site is accurate and up-to-date.

Now, according to this blog post

The Codex models are descendants of our GPT-3 models that can understand and generate code. Their training data contains both natural language and billions of lines of public code from GitHub

code-davinci-002 is a codex model. So, ChatGPT might be a descendent of GPT-3. I don't know what "descendent" exactly means here. Does it mean just fine-tuned or maybe a modified version?

Here they write that text-davinci-003 is the most capable GPT-3 model. Based on the information above, text-davinci-002 is an InstructGPT model based on code-davinci-002.

Here they write

We then use this data to fine-tune GPT-3.

The resulting InstructGPT models are much better at following instructions than GPT-3

So, InstructGPT models are fine-tuned GPT-3 models. That most likely implies that text-davinci-002 is a GPT-3 model and the only thing that changes is how it was trained. However, they also write

Our labelers prefer outputs from our 1.3B InstructGPT model over outputs from a 175B GPT-3 model, despite having more than 100x fewer parameters.

So, there isn't just one GPT3 model. However, the original GPT3 model had 175 billion parameters.

I still need to read the InstructGPT and GPT3 papers. Once I've done that, I may have more useful/concrete info, then I will update this answer.