I'm basically trying to replicate the processed used to create Chat GPT: enter image description here

Am I supposed to backpropagate? How can I do that when these aren't really errors, but rather ranking several response? Can I use a 1-10 rating system where 10 is a perfect response instead so I have something closer to an error signal?

Also I have to rank/rate each output as a whole, but the final layer of the model are token neurons. Do I somehow connect my ranking data to the layer right before the final layer? Is the reward model supposed to be completely separate from the transformer with no direct edges/weight connections? Do I update the transformer model using the Bellman equations?

I am shocked I can't find a tutorial or really any information about this crucial component given that this is the strategy used to make Chat GPT according to OpenAI. Is there something I'm missing that someone could direct me towards? All I can find is information about fine-tuning transformers and RLHF separately, but not combined.



You must log in to answer this question.

Browse other questions tagged .