0
$\begingroup$

For a transformer decoder, how exactly are K, Q, and V for each decoding step?

Assume my input prompt is "today is a" (good day).

At t= 0 (generation step 0): K, Q, and V are the projections of the sequence ("today is a") Then say the next token generated is "good".

At t=1 (generation step 1): Which one is true:

  1. K, Q, and V are the projections of the sequence ("today is a good")
  2. K, Q, are the projections of the sequence ("today is a"), and V is the projection of the sequence ("good")?
Improve this question
$\endgroup$
3
  • 3
    $\begingroup$ Please see if the explanation here helps you in any way. In short, during the forward pass, K, Q, and V are projections of the input sequence ("today is a"). The output of the forward pass is "good". $\endgroup$
    – Robin van Hoorn
    Commented May 9, 2023 at 7:03
  • $\begingroup$ @RobinvanHoorn, I see, and thanks. My question is the next next time step (t+2), so now your entire sequence is "today is a good", do you just repeat the same thing (with K, Q, V as the projections of the "today is a good" sequence?) $\endgroup$
    – wrek
    Commented May 10, 2023 at 2:33
  • 1
    $\begingroup$ your question was (and the question still only mentions) t=1. If you again want to predict a next word, then K,Q and V are again projections of the input sequence ("today is a good") and then the output becomes (i.e.) "day". $\endgroup$
    – Robin van Hoorn
    Commented May 10, 2023 at 9:50

1 Answer 1

Reset to default
2
$\begingroup$

(This type of) autoregressive LLM always works by predicting one next token based on a series of previous tokens. First you run the model with input "today is a" and the prediction is "good". Then you run the model with input "today is a good" and the prediction is "day", and so on. Each token is predicted by running the entire model from start to finish on its previous input.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .