I've been trying to understand how the modern LLMs use PPO for fine-tuning. In the PPO algorithm, one has to compute advantages, which are then used for either increasing or decreasing action's probability by following gradient; but what's the action space in this problem?

There are 2 straightforward options I see for the meaning of action $a_t$ in $s_t$:

  • Choosing next token for the given prompt. In this case, $s_{t+1}$, would be equal to prompt + next token.
  • Generating whole completion for the given prompt. In this case, all episodes will be of the constant length one.

I've read various papers on this and to me is not clear what's typically used and why. In the InstructGPT they mention:

The environment is a bandit environment which presents a random customer prompt and expects a response to the prompt. Given the prompt and response, it produces a reward determined by the reward model and ends the episode.

However, in the Secrets of RLHF in Large Language Models Part I: PPO they mention that $a_t$ represents next token. In this case, I fail to understand how our reward model $r_{\theta}$, which was trained on the computing scalar for prompts $x$ and whole completions $y$, would be able to compute immediate reward after taking action $a_t$ (next token). Wouldn't those kinds of incomplete prompts be far outside the distribution of the training data?

Also, in the Llama 2 paper, they optimize following objective:

$$\arg \max_{\pi} \mathbb{E}_{p \sim D, g \sim \pi}[R(g | p)]$$

But what is meant by generations $g$? If it's once again just the next token, does not it suffer from the same problem with reward model?



You must log in to answer this question.