2
$\begingroup$

I am confused about the difference betwen fine runing and rlhf for llm. When to use what? I know RLHF need to creating a reward model which at furst rates responses to align the responses to the human preferences and afterward using this reward model to fine-tune.

But if thats the case, when fine-tuning is relevant anymore. When to use what?

$\endgroup$

2 Answers 2

2
$\begingroup$

RLHF is just one possibility of fine-tuning for generative LLMs, which is used to align an LLM to human tastes.

However, you could just create a bunch of great data, and fine-tune (take a pretrained models and just slightly adjust it) on this high quality data, without any RLHF involved (see for example what phi-1 has done)

$\endgroup$
0
$\begingroup$

So finetuning (on a supervised dataset) is a) extremely expensive (in terms of data collection and annotation, as well as GPU costs). RLHF gets around this by training a reward model as well as a policy model. The reward model provides a signal to the generated outputs, and the policy model helps guide the LLM towards correct/aligned decoding during generation time.

Sources:

  1. https://www.assemblyai.com/blog/how-rlhf-preference-model-tuning-works-and-how-things-may-go-wrong/
  2. https://arxiv.org/pdf/2204.05862.pdf
$\endgroup$
1
  • $\begingroup$ Gpu cost is probably the least concern for fine tuning. $\endgroup$
    – John Jiang
    Jan 15 at 3:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .