Skip to main content

# Questions tagged [rlhf]

For questions related to RLHF: Reinforcement Learning from Human Feedback

10 questions
Filter by
Sorted by
Tagged with
4 votes
1 answer
939 views

### Why do we need RL in RLHF? [closed]

In RLHF, the reward function is a neural network. This means we can compute its gradients cheaply and accurately through backpropagation. Now, we want to find a policy that maximizes reward (see https:...
• 204
2 votes
2 answers
567 views

### What is the difference betwen fine runing and rlhf for llm?

I am confused about the difference betwen fine runing and rlhf for llm. When to use what? I know RLHF need to creating a reward model which at furst rates responses to align the responses to the ...
• 353
1 vote
0 answers
41 views

### When reading on RLHF, I came across this formula but can't break it down

What is the exact meaning of this expression? I'm unsure on the notation. I believe E[R(s)] is expected value of reward of state s, but I'm unsure what the subscript under the E means.
1 vote
0 answers
223 views

### Negative KL-divergence RLHF implementation

I am struggling to understand one part of the FAQ of the transformer reinforcement learning library from HuggingFace: What Is the Concern with Negative KL Divergence? If you generate text by purely ...
0 votes
0 answers
30 views

### What's the action space in RLHF for LLM?

I've been trying to understand how the modern LLMs use PPO for fine-tuning. In the PPO algorithm, one has to compute advantages, which are then used for either increasing or decreasing action's ...
• 149
0 votes
1 answer
57 views

### understanding the distribution shift problem in direct preference optimization (DPO)

I'm having trouble understanding this paragraph of the DPO paper: Why does it matter so much that the preference data distribution aligns with the reference model output distribution? My ...
0 votes
0 answers
111 views

### Why is DPO necessary when the exact solution for RLHF/PPO is already available?

I came across an interesting paper on improving RLHF (essentially getting rid of the RL part), highly recommended by Andrew Ng, and receiving already 140 citations in 1 month (though the preprint ...
• 101
0 votes
0 answers
50 views

### When using Reinforcement Learning with Human Feedback to train a transformer, how do I propagate the feedback through the transformer?

I'm basically trying to replicate the processed used to create Chat GPT: Am I supposed to backpropagate? How can I do that when these aren't really errors, but rather ranking several response? Can I ...
0 votes
0 answers
22 views

### What is the policy model in RLHF for LLMs?

What is the policy model doing explicitly in an LLM with RLHF setup? From my understanding, LLMs generate in a way that is no different from any of their predecessors: beam search decoding, ...
0 votes
0 answers
78 views

### Can pretraining be continued after RLHF?

Assume you have a pretrained transformer language model (M1) which already underwent reinforcement learning by human feedback (M2). I assume that it is in principle possible to continue the ...