Timeline for InstructGPT: What is the sigma in the loss function and why $\log(\cdot)$ is being used?
Current License: CC BY-SA 4.0
9 events
when toggle format | what | by | license | comment | |
---|---|---|---|---|---|
Feb 7, 2023 at 19:15 | comment | added | schreon | @itisyeetimetoday first, the loss is based on a comparison of A and B, so asking for only calculating the "loss of B" does not make much sense. You would rather say "the loss of the comparison of A and B, where B should be higher than A". So if the human annotator said that (given the instruction $x$) the response B is preferred over A, then you assign $y_w = B$ and $y_l = A$. Then you feed $(x, y_w)$ into your reward model $r_\theta$, resulting in one scalar value (the value of the output neuron of your reward model). Repeat this for $(x, y_l)$ and you can calculate that quantity :-) | |
Feb 7, 2023 at 3:42 | comment | added | itisyeetimetoday | @schreon Thank you for your response. I'm struggling to understand the quantity inside the sigmoid function: 𝑟𝜃(𝑥,𝑦𝑤)−𝑟𝜃(𝑥,𝑦𝑙). If I am comparing two responses, A and B, and want to calculate the loss of B, if B is ranked higher than A, how is 𝑟𝜃(𝑥,𝑦𝑤)−𝑟𝜃(𝑥,𝑦𝑙) calculated? | |
Feb 1, 2023 at 8:51 | history | edited | schreon | CC BY-SA 4.0 |
fixed some notation mistakes regarding the reward function
|
Feb 1, 2023 at 8:44 | comment | added | schreon | NathanB and @respectful: alright, I expanded the answer in order to give a full explanation of the loss formula. This is based on the linked guide, the original InstructGPT paper, as well as my personal understanding/interpretation of the material. | |
Feb 1, 2023 at 8:39 | history | edited | schreon | CC BY-SA 4.0 |
expanded the answer based on whas is written in the linked guide, the InstructGPT paper and my personal understanding/interpretation
|
Jan 30, 2023 at 17:22 | comment | added | respectful | To improve this answer could you summarize the specific information from the source that answers the question. This link could become invalid in the future. So by extracting the content that answers the question makes it a stronger and more durable answer for those in the future. | |
Jan 29, 2023 at 11:27 | comment | added | Nathan G | Thank you. Any idea why it is actually needed here? | |
S Jan 27, 2023 at 8:55 | review | First answers | |||
Jan 30, 2023 at 17:22 | |||||
S Jan 27, 2023 at 8:55 | history | answered | schreon | CC BY-SA 4.0 |