The probability of preferring one trajectory/response (say $A$) over the other (say $B$) is given by,
$$ \begin{aligned} P(A>B) &= \frac{\exp(r_\theta(A))}{\exp(r_\theta(A))+\exp(r_\theta(B))}\\ &= \frac{1}{1+\exp(-(r_\theta(A)-r_\theta(B)))}\\ &= \sigma(r_\theta(A)-r_\theta(B)) \end{aligned} $$
where $\sigma(x) = \frac{1}{1+e^{-x}}$ is the sigmoid function. This follows from the Bradley-Terry model for estimating score functions from pairwise preferences. If say $y_w$ and $y_l$ are the responses from the model (augmented with the query $x$), and we prefer $y_w$ over $y_l$, then $\mu(y_w)=1; \mu(y_l)=0$. We can use the following cross-entropy loss function and substitute for the probability from above equation to train our reward model:
$$ \begin{aligned} L &= -\mu(y_w) \log(P(y_w>y_l)) + \mu(y_l) \log(P(y_l>y_w))\\ &= -\log(P(y_w>y_l))\\ &= -\log(\sigma(r_\theta(x,y_w)-r_\theta(x,y_l))) \end{aligned} $$
In InstructGPT, the model is made to generate $K$ responses. So we can have $K\choose2$ pairs of comparisons that we can make. Example if the model generates four responses, $A, B, C, D$ and our ranking is $B>C>D>A$, then there are ${4\choose2}=6$ comparisons possible: $B>C$, $B>D$, $B>A$, $C>D$, $C>A$ and $D>A$. The loss function in this case reduces to,
$$ L = - \frac{1}{K\choose2} E_{(x,y_w,y_l) \in \mathcal{D}}\Big[\log(\sigma(r_\theta(x,y_w)-r_\theta(x,y_l)))\Big] $$
Hope this helps. I have created a blog post to explain RLHF in conversational AI models here if you want to understand better.