1
$\begingroup$

If I have a negative example from a large language model like

{"prompt": "What is the capital of Italy?", "completion": "Milan"}

Is there a way for me to use this negative (incorrect) example to finetune an LLM using SFT so that the model is less likely to produce this answer? The correct answer to this question is "Rome".

Improve this question
$\endgroup$
1
  • $\begingroup$ yes but you have to be very carefull... you can use a negative CCE loss, thus maximizing it using that answer as correct answer (it pretty much downweights that answers)... however, you cannot control which other answer is upweighted, so not really sure that this would help $\endgroup$
    – Alberto
    Commented Apr 30 at 22:39

1 Answer 1

Reset to default
1
$\begingroup$

Yes, but I have not seen it done with purely negative samples. Often alignment is done using a positive-negative pair. This is done in RLHF, DPO, and ORPO.

There are obviously methods that use solely positive samples so it is mathematically tractable to right this in reverse, using solely negative samples. Often the problem with this is that the space of what you don't want is combinatorically large. Methods that use positive-negative pairs work because the positive and negative answers are syntax and grammar similar. This is better explained by the ORPO paper where they show that fine-tuning on positive-only samples actually increases occurrence of negative samples.

An example of this:

(Prompt): Why is the sky green?

(Positive): The sky is not green.

(Negative): The sky is green because...

These answers start off quite similar and choose some of the same tokens. Therefore negative-only could actually cause the model to unlearn positive answers as well.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .