Skip to main content
9 events
when toggle format what by license comment
Feb 7, 2023 at 19:15 comment added schreon @itisyeetimetoday first, the loss is based on a comparison of A and B, so asking for only calculating the "loss of B" does not make much sense. You would rather say "the loss of the comparison of A and B, where B should be higher than A". So if the human annotator said that (given the instruction $x$) the response B is preferred over A, then you assign $y_w = B$ and $y_l = A$. Then you feed $(x, y_w)$ into your reward model $r_\theta$, resulting in one scalar value (the value of the output neuron of your reward model). Repeat this for $(x, y_l)$ and you can calculate that quantity :-)
Feb 7, 2023 at 3:42 comment added itisyeetimetoday @schreon Thank you for your response. I'm struggling to understand the quantity inside the sigmoid function: 𝑟𝜃(𝑥,𝑦𝑤)−𝑟𝜃(𝑥,𝑦𝑙). If I am comparing two responses, A and B, and want to calculate the loss of B, if B is ranked higher than A, how is 𝑟𝜃(𝑥,𝑦𝑤)−𝑟𝜃(𝑥,𝑦𝑙) calculated?
Feb 1, 2023 at 8:51 history edited schreon CC BY-SA 4.0
fixed some notation mistakes regarding the reward function
Feb 1, 2023 at 8:44 comment added schreon NathanB and @respectful: alright, I expanded the answer in order to give a full explanation of the loss formula. This is based on the linked guide, the original InstructGPT paper, as well as my personal understanding/interpretation of the material.
Feb 1, 2023 at 8:39 history edited schreon CC BY-SA 4.0
expanded the answer based on whas is written in the linked guide, the InstructGPT paper and my personal understanding/interpretation
Jan 30, 2023 at 17:22 comment added respectful To improve this answer could you summarize the specific information from the source that answers the question. This link could become invalid in the future. So by extracting the content that answers the question makes it a stronger and more durable answer for those in the future.
Jan 29, 2023 at 11:27 comment added Nathan G Thank you. Any idea why it is actually needed here?
S Jan 27, 2023 at 8:55 review First answers
Jan 30, 2023 at 17:22
S Jan 27, 2023 at 8:55 history answered schreon CC BY-SA 4.0