In the official blog post about ChatGPT from OpenAI, there is this paragraph explaining how ChatGPT model was trained:
We trained this model using Reinforcement Learning from Human Feedback (RLHF), using the same methods as InstructGPT, but with slight differences in the data collection setup. We trained an initial model using supervised fine-tuning: human AI trainers provided conversations in which they played both sides—the user and an AI assistant. We gave the trainers access to model-written suggestions to help them compose their responses. We mixed this new dialogue dataset with the InstructGPT dataset, which we transformed into a dialogue format.
Especially this part:
We trained an initial model using supervised fine-tuning
My question is about the said initial model, is it some new model that has been trained from scratch or is it a GPT-3 model that has been fine tuned for specific tasks resulting in GPT-3.5 series ?
On the other hand, from the InstructGPT blog post, it is clearly stated that:
To make our models safer, more helpful, and more aligned, we use an existing technique called reinforcement learning from human feedback (RLHF). On prompts submitted by our customers to the API,our labelers provide demonstrations of the desired model behavior, and rank several outputs from our models. We then use this data to fine-tune GPT-3.
So does this mean that GPT-3.5 series models (and consequently ChatGPT) are fine-tuned from GPT-3 base model ?