Skip to main content

Questions tagged [rlhf]

For questions related to RLHF: Reinforcement Learning from Human Feedback

Filter by
Sorted by
Tagged with
4 votes
1 answer
939 views

Why do we need RL in RLHF? [closed]

In RLHF, the reward function is a neural network. This means we can compute its gradients cheaply and accurately through backpropagation. Now, we want to find a policy that maximizes reward (see https:...
DeltaIV's user avatar
  • 204
2 votes
2 answers
567 views

What is the difference betwen fine runing and rlhf for llm?

I am confused about the difference betwen fine runing and rlhf for llm. When to use what? I know RLHF need to creating a reward model which at furst rates responses to align the responses to the ...
Exploring's user avatar
  • 353
1 vote
0 answers
41 views

When reading on RLHF, I came across this formula but can't break it down

What is the exact meaning of this expression? I'm unsure on the notation. I believe E[R(s)] is expected value of reward of state s, but I'm unsure what the subscript under the E means.
Ryan Marr's user avatar
1 vote
0 answers
223 views

Negative KL-divergence RLHF implementation

I am struggling to understand one part of the FAQ of the transformer reinforcement learning library from HuggingFace: What Is the Concern with Negative KL Divergence? If you generate text by purely ...
probably45's user avatar
0 votes
0 answers
30 views

What's the action space in RLHF for LLM?

I've been trying to understand how the modern LLMs use PPO for fine-tuning. In the PPO algorithm, one has to compute advantages, which are then used for either increasing or decreasing action's ...
Druudik's user avatar
  • 149
0 votes
1 answer
57 views

understanding the distribution shift problem in direct preference optimization (DPO)

I'm having trouble understanding this paragraph of the DPO paper: Why does it matter so much that the preference data distribution aligns with the reference model output distribution? My ...
Ivy Cao's user avatar
0 votes
0 answers
111 views

Why is DPO necessary when the exact solution for RLHF/PPO is already available?

I came across an interesting paper on improving RLHF (essentially getting rid of the RL part), highly recommended by Andrew Ng, and receiving already 140 citations in 1 month (though the preprint ...
John Jiang's user avatar
0 votes
0 answers
50 views

When using Reinforcement Learning with Human Feedback to train a transformer, how do I propagate the feedback through the transformer?

I'm basically trying to replicate the processed used to create Chat GPT: Am I supposed to backpropagate? How can I do that when these aren't really errors, but rather ranking several response? Can I ...
Austin Capobianco's user avatar
0 votes
0 answers
22 views

What is the policy model in RLHF for LLMs?

What is the policy model doing explicitly in an LLM with RLHF setup? From my understanding, LLMs generate in a way that is no different from any of their predecessors: beam search decoding, ...
information_interchange's user avatar
0 votes
0 answers
78 views

Can pretraining be continued after RLHF?

Assume you have a pretrained transformer language model (M1) which already underwent reinforcement learning by human feedback (M2). I assume that it is in principle possible to continue the ...
Hans-Peter Stricker's user avatar