0
$\begingroup$

When an LLM creates an output, it seemingly has no way to check if its output was valid. Therefore it wouldn't be able to back-propagate any changes to the weights is used to create that output.

Right now, I suspect that all weight modification is done by training on input data, as that can (generally) be assumed to be human and valid.

But perhaps it does have some way of checking if its output was good or not, and being modified based off of it. For instance, there is the thumb up and down button on ChatGPT which could be used for that.

Do LLM's modify their neural weights based off of their own answers? If so, how?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I will base my answer on GPT-2, as most LLMs are not too different in their architecture.

The output of the transformer in an LLM is not text or tokens; it is a categorical distribution over the space of next possible tokens.

To calculate loss of the transformer, you (commonly) take negative log likelihood. If the training data says the next token was $i$ and the transformer's output was a vector $v$, then we have this loss. $$L = -\log(v_i)$$ Backpropagating this encourages the transformer to assign higher confidence to the correct token, according to data.

Reagarding the 'thumbs up' feature of ChatGPT, this is probably an auxiliary loss. OpenAI is known to use Reinforcement Learning to tune ChatGPT, in addition to the negative log likelihood which would've been used during pre-training. How exactly this is calculated, we don't know; ChatGPT is closed source.

Improve this answer
$\endgroup$
2
  • $\begingroup$ "If the training data says the next token was 𝑖" how would this apply to text it outputs? This makes sense for training with user generated text. $\endgroup$
    – Seph Reed
    Commented Jul 13, 2023 at 4:08
  • 1
    $\begingroup$ @SephReed Most LLMs are trained on human data, so full text generated by the network is not taken into account, only the token probabilities. In RLHF, the full texts are used for fine-tuning. A human is able to assign scores to a token sequence that the LLM generates. 'credit' is then assigned to each of the tokens, and the transformer adapts its token probabilities to maximise score. You could have a look at InstructGPT which is an example of this. $\endgroup$
    – Venna Banana
    Commented Jul 13, 2023 at 4:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .