InstructGPT: What is the sigma in the loss function and why $\log(\cdot)$ is being used?
$$ \operatorname{loss}(\theta) = -\frac{1}{\binom{K}{2}}E_{(x,y_w,y_l)\sim D}[\log(\sigma(r_{\theta}(x, y_w) - r_{\theta}(x, y_l)))] $$
The equation was taken from the InstructGPT paper.
According to this guide, the sigma in this formula refers to the sigmoid activation function. The guide does not tell exactly why the sigmoid function is used here, so I will try to give a full explanation of how this loss formulation works (page 8, formula 1 in the InstructGPT paper):
$\text{loss}(\theta)=-\frac{1}{\binom{K}{2}}E_{(x,y_w,y_l) \sim D} [log(\sigma(r_\theta(x,y_w)-r_\theta(x,y_l)))]$
In the following I will use the notation from the paper:
$x$ refers to the given instruction
($y_w$, $y_l$) refers to a pair of responses out of the list of responses which a human ranked based on their preference
$y_w$ refers to the response that is preferred over the other, lesser preferred response $y_l$
$r$ refers to the reward model
$\theta$ refers to the trainable parameters of that reward model
$\sigma$ refers to the sigmoid activation function.
If you interpret $\sigma(r_\theta(x, y_w) - r_\theta(x, y_l))$ as the probability that the reward model assigns a higher reward to the preferred response $y_w$ than to the lesser preferred response $y_l$, the formula makes total sense. How do you maximize a probability? Exactly, by minimizing the negative log of that probability (this is called negative log likelihood). Hence the minus sign in front of the formula and log function around the sigmoid.
In order to elaborate a bit more on that: If the reward model is working fine, then $r_\theta(x, y_w)$ will be a very large positive number and $r_\theta(x, y_l)$ will be a much lower number (maybe even a negative number). The difference $r_\theta(x, y_w) - r_\theta(x, y_l)$ will then be a very large positive number. And the sigmoid of a very large positive number approaches $1$. In that case, everything is according to plan and the loss will be very small (close to zero). However, if the reward model is failing, the assigned reward $r_\theta(x, y_w)$ might be much smaller than $r_\theta(x, y_l)$. Hence, the difference $r_\theta(x, y_w) - r_\theta(x, y_l)$ will be a (possibly very large) negative number. Take the sigmoid of that and you get a value that approaches $0$ (thus, the probability that the reward model assigns a higher reward to the preferred response will be small). As we are trying to maximize a probability by minimizing the negative log likelihood, we get a large loss in that case.
As there will be a varying number of ranked responses for each instruction in one batch ("between $K = 4$ and $K = 9$ responses"), the losses of those pairwise comparisons must be weighted, so that each instruction has the same impact on the gradient update, no matter how many responses the humans have been presented for each instruction. The number of pairwise comparisons out of $K$ is $\binom{K}{2}$.
In order to wrap it up: By minimizing the loss described in the paper, the reward model gets incentivized to assign a large positive reward to responses the (hopefully adequately paid cough) humans in front of their computers consider to be very good responses and very large negative rewards to responses which those humans consider to be very bad. And this is exactly what is desired in order to fine-tune an LLM according to human preference using reinforcement learning.
The probability of preferring one trajectory/response (say $A$) over the other (say $B$) is given by,
$$ \begin{aligned} P(A>B) &= \frac{\exp(r_\theta(A))}{\exp(r_\theta(A))+\exp(r_\theta(B))}\\ &= \frac{1}{1+\exp(-(r_\theta(A)-r_\theta(B)))}\\ &= \sigma(r_\theta(A)-r_\theta(B)) \end{aligned} $$
where $\sigma(x) = \frac{1}{1+e^{-x}}$ is the sigmoid function. This follows from the Bradley-Terry model for estimating score functions from pairwise preferences. If say $y_w$ and $y_l$ are the responses from the model (augmented with the query $x$), and we prefer $y_w$ over $y_l$, then $\mu(y_w)=1; \mu(y_l)=0$. We can use the following cross-entropy loss function and substitute for the probability from above equation to train our reward model:
$$ \begin{aligned} L &= -\mu(y_w) \log(P(y_w>y_l)) + \mu(y_l) \log(P(y_l>y_w))\\ &= -\log(P(y_w>y_l))\\ &= -\log(\sigma(r_\theta(x,y_w)-r_\theta(x,y_l))) \end{aligned} $$
In InstructGPT, the model is made to generate $K$ responses. So we can have $K\choose2$ pairs of comparisons that we can make. Example if the model generates four responses, $A, B, C, D$ and our ranking is $B>C>D>A$, then there are ${4\choose2}=6$ comparisons possible: $B>C$, $B>D$, $B>A$, $C>D$, $C>A$ and $D>A$. The loss function in this case reduces to,
$$ L = - \frac{1}{K\choose2} E_{(x,y_w,y_l) \in \mathcal{D}}\Big[\log(\sigma(r_\theta(x,y_w)-r_\theta(x,y_l)))\Big] $$
Hope this helps. I have created a blog post to explain RLHF in conversational AI models if you want to understand better.
