I'm curious about something in the decoder part of the Transformers architecture. From what I understand, the Keys and Values come from the output of the encoder part of the Transformer. I understand that we have as many queries, keys and values as we have input tokens (actually if it is multiheaded attention we may have more (i.e. 10 heads means 10 queries for each word)).

In the decoder part, we basically match (by doing a scalar product) a query that the output "asks" to those keys and values to know which part of the input the decoder should look at.

However, if I'm not mistaken, there are as many queries as there are previously outputted output tokens. Therefore as many outputs of the multi-head attention as there are previously outputted output tokens.

How / when does this dimensionality that depends on the current amount of output tokens disappear? It has to disappear since "at the end" there is a standard feed-forward network which size is fixed* and cannot depend on the number of output tokens. I'm talking about inference when one token at a time is generated but the transformer still inputs to the decoder all the previous generated output tokens. I'm not sure about that last part especially since it says "Outputs (shifted right)" in the paper. Maybe it only inputs a fixed number of previously outputted tokens? But that feels wrong.

* or maybe each of those outputs is fed through the last feed forward network but then how is a word determined with those $n_{previously\_outputted\_tokens}$ outputs of this last feed forward network?


1 Answer 1


Yes, if you input into a transformer $n$ input embeddings, you will receive $n$ output embeddings. But, when making predictions, the feed forward layer will only use one of those. For example, when a decoder model (e.g., GPT-2) predicts the next token, it will use the last embedding, and BERT has a special [CLS] token that is supposed to pool information from the entire input.

  • $\begingroup$ So in the multi-head attention connecting the encoder and the decoder, only one of the output embedding is used? But in the masked multi-head attention they are all used? $\endgroup$ Jan 26 at 18:11
  • $\begingroup$ No usually, only one token is used for each prediction for MLM as well. e.g., with BERT, the [CLS] token embedding is used (during pretraining) for the next sentence prediction. The "pooling" happens implicitly as a result of the pretraining task. $\endgroup$ Jan 26 at 19:07
  • $\begingroup$ I'm sorry I'm getting confused because in the paper they explicitly say: "At each step the model is auto-regressive, consuming the previously generated symbols as additional input when generating the next". Symbols is plural so surely they use more than one token right? $\endgroup$ Jan 27 at 9:27
  • $\begingroup$ "At each step" means at each (next token) prediction. Each step can use a different token, but each prediction only uses a single token. $\endgroup$ Jan 27 at 15:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .