Skip to main content

Questions tagged [rlhf]

For questions related to RLHF: Reinforcement Learning from Human Feedback

8 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
1 vote
0 answers

When reading on RLHF, I came across this formula but can't break it down

What is the exact meaning of this expression? I'm unsure on the notation. I believe E[R(s)] is expected value of reward of state s, but I'm unsure what the subscript under the E means.
Ryan Marr's user avatar
1 vote
0 answers

Negative KL-divergence RLHF implementation

I am struggling to understand one part of the FAQ of the transformer reinforcement learning library from HuggingFace: What Is the Concern with Negative KL Divergence? If you generate text by purely ...
probably45's user avatar
0 votes
0 answers

What's the action space in RLHF for LLM?

I've been trying to understand how the modern LLMs use PPO for fine-tuning. In the PPO algorithm, one has to compute advantages, which are then used for either increasing or decreasing action's ...
Druudik's user avatar
  • 149
0 votes
1 answer

understanding the distribution shift problem in direct preference optimization (DPO)

I'm having trouble understanding this paragraph of the DPO paper: Why does it matter so much that the preference data distribution aligns with the reference model output distribution? My ...
Ivy Cao's user avatar
0 votes
0 answers

Why is DPO necessary when the exact solution for RLHF/PPO is already available?

I came across an interesting paper on improving RLHF (essentially getting rid of the RL part), highly recommended by Andrew Ng, and receiving already 140 citations in 1 month (though the preprint ...
John Jiang's user avatar
0 votes
0 answers

When using Reinforcement Learning with Human Feedback to train a transformer, how do I propagate the feedback through the transformer?

I'm basically trying to replicate the processed used to create Chat GPT: Am I supposed to backpropagate? How can I do that when these aren't really errors, but rather ranking several response? Can I ...
Austin Capobianco's user avatar
0 votes
0 answers

What is the policy model in RLHF for LLMs?

What is the policy model doing explicitly in an LLM with RLHF setup? From my understanding, LLMs generate in a way that is no different from any of their predecessors: beam search decoding, ...
information_interchange's user avatar
0 votes
0 answers

Can pretraining be continued after RLHF?

Assume you have a pretrained transformer language model (M1) which already underwent reinforcement learning by human feedback (M2). I assume that it is in principle possible to continue the ...
Hans-Peter Stricker's user avatar