2
$\begingroup$

My rough understanding of RLHF as used for ChatGPT in a nutshell is this:

  1. A reward model is trained using comparisons of different responses to the same prompt. Human trainers rank these responses based on quality.

  2. The reward model is a neural network that learns to predict these human rankings. It essentially learns the "policy" that human trainers use to rank responses.

  3. An initial policy, which is a language model, is fine-tuned using Proximal Policy Optimization (PPO) with the reward model providing the reward signal. This process is iterative, with the policy and reward model being updated alternately.

  4. The policy is then used to generate responses to prompts. The reward model assesses these responses and provides a reward signal, which is used to further fine-tune the policy, i.e. the language model.

My main question is the first one, the others are just for giving context:

1. What's the architecture and size of the neural-network-based reward model?

  1. Is it pretrained, too? Is it possibly another pretrained (foundational) language model?

  2. By how many samples labelled by human trainers is the reward model trained?

  3. By how many prompts and rewarded completions is the language model trained later? (Which prompts, by the way?)

These numbers I'd like to compare with the numbers of pretrained ChatGPT:

  • Transformer-based ChatGPT has 175 billion weights.

  • It was pretrained on 500 GB of text data, distributed over an unknown number of "documents" (from single tweets to the Holy Bible) with roughly 500B tokens over all. During training ChatGPT was exposed to a multiple of 500B samples (assuming that all 500B tokens were used for training).

I assume that during RLHF foundational ChatGPT was exposed to a much smaller number of prompts to complete (and to be rewarded).

$\endgroup$
6
  • $\begingroup$ It seems you're asking many questions here, which makes this post too broad. If you're asking many questions, please, pick one, then ask the others in separate posts, unless they are really sub-questions that would answer the main question. Once you've picked your main question, please, put it in the title for clarity. $\endgroup$
    – nbro
    Jul 9, 2023 at 23:14
  • 1
    $\begingroup$ @nbro: Is it better now? $\endgroup$ Jul 10, 2023 at 9:47
  • $\begingroup$ @nbro: Can you imagine why this question is not so well received: viewed only rarely and even downvoted? Is it ill-posed or trivial? $\endgroup$ Jul 11, 2023 at 7:53
  • 1
    $\begingroup$ There has been a lot of hype around ChatGPT, LLM, etc. As you probably have noticed, hype comes and goes. I think your questions are interesting, more interesting than those like "Why can't ChatGPT do math?", or stuff like that, which are boring to me. You're asking technical/theoretical questions, which seems to be suitable for our site. Last time I checked, there was no official ChatGPT paper. The only official details that we have about ChatGPT are only in the OpenAI posts on the topic. So, maybe this info is not yet available. I'd need to check there. The InstructGPT paper may be useful $\endgroup$
    – nbro
    Jul 12, 2023 at 12:31
  • 1
    $\begingroup$ Anyway, maybe your post can still be simplified. To me, it still seems you're asking different questions about the reward model. I don't know if removing all but 1 question will help to attract more people. To me, your post is still asking the question: "Can you please provide more details about the reward model used in ChatGPT?" and then you ask some questions with specific details you're interested in. Maybe it's fine to keep it like that. $\endgroup$
    – nbro
    Jul 12, 2023 at 12:41

1 Answer 1

1
+100
$\begingroup$

If you haven't already, I would recommend a careful reading of OpenAI's paper on InstructGPT. This was their publication from last year regarding how they applied RLHF to GPT-3, the precursor of ChatGPT.

The appendix provides information on the reward model and the RLHF training data. For example,

For the reward models and value functions, the unembedding layer of the original model is replaced with a projection layer to output a scalar value.

The final reward model was initialized from a 6B GPT-3 model that was fine-tuned on a variety of public NLP datasets (ARC, BoolQ, CoQA, DROP, MultiNLI, OpenBookQA, QuAC, RACE, and Winogrande).

and,

We train all the RL models for 256k episodes. These episodes include about 31k unique prompts, after filtering out prompts with PII and deduplication based on common prefixes.

If you want to know what ChatGPT does specifically, you might have to ask someone who works there. It's not public information.

$\endgroup$
1
  • $\begingroup$ There does not have to be anything published given that RLHF is by now done by a ton of open source projects. $\endgroup$
    – TomTom
    Jul 14, 2023 at 9:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .