What does attending to future tokens mean? From my understanding, the transformer model works by inputting a prompt and predicting the next word in a sequence and this process just keeps repeating while attending to the words from the prompt and the already generated words. However, the explanations for transformers all mention the prevention of attending to future tokens while generating the next word in a sequence. How do these models attend to a word that hasn’t been generated yet?


1 Answer 1


Most of the recent hype around transformers is about the decoder-only transformer architecture. This architecture is different from the originally introduced traditional transformer which features both an encoder and a decoder. A transformer only attends to future tokens in the encoder. Hence, models such as chatGPT, GPT-4 etc, do not attend to future tokens.

The traditional transformer (featuring an encoder and a decoder) was introduced for the task of language translation. Hence, you would input a prompt of language A into the encoder and the decoder would output that same prompt in language B. In language translation tasks, you need to attend to future tokens. In different languages, pieces of sentences are ordered differently. You cannot simply translate each word at the same place.

Hence, the encoder of the traditional transformer model attends to future tokens. It can do so because the whole input prompt is fed into the encoder at once. You have to think of the transformer model like a standard Multi-Layer Perceptron (MLP). An MLP calculates the output for the whole input at once. The transformer does the same. It uses information from the complete input prompt to calculate what parts of the input prompt to attend to. This can be at the start of the prompt, but also at the end of the prompt. The decoder then uses this information, as well as the previously translated words, to generate the next translated word/token.

Lots of confusion is currently occurring surrounding transformers because it is not clearly explained that 'the transformer' comes in different forms. The most popular transformer right now is the decoder-only transformer (see full explanation here). It is much easier to understand how the traditional transformer works by first understanding how the decoder-only transformer works. So I'd recommend reading the linked post and then thinking again about whether you now understand the answer to this question. I expect you to easily answer your current question after you complete understand the decoder-only transformer architecture.

  • $\begingroup$ If this answer has sufficiently answered your question, please consider assigning it the 'green checkmark' such that future visitors of the site can better find what they are looking for :D $\endgroup$ Commented Oct 27, 2023 at 13:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .