How is InstructGPT a fine-tuned version of GPT-3 and at the same time has fewer parameters than the original GPT3?

Question

I am reading the paper "Training language models to follow instructions with human feedback"

It says:

Our labelers provide demonstrations of the desired behavior on the input prompt distribution (see Section 3.2 for details on this distribution). We then fine-tune a pretrained GPT-3 model on this data using supervised learning.

The paper also says:

On our test set, outputs from the 1.3B parameter InstructGPT model are preferred to outputs from the 175B GPT-3, despite having over 100x fewer parameters.

I am not able to understand how the aligned model is a fine tune of GPT-3 using supervised learning (and other steps associated with reinforcement learning) and at the same time the aligned model has fewer parameters than the original model.

Can someone give me a hint on the subject?

Answer 1

They say “a pretrained GPT-3” model, emphasis on “a” implying one of many, rather than “the”.

I believe they simply repeat the process with various parameters scales of the pretrained GPT-3, comparing the performance to GPT-3 175B throughout to see if there are parameter efficiency gains.

They note that even the 1.3B instructGPT version outperforms the 175B GPT-3 version.