How does a LLM (transformer) pick words from its vocabulary?

Question

I have a very rough understanding of the "attention/self attention" mechanism of transformer models and how this can be used to process a set of word vectors provided as an input/prompt to the encoder of a network and how this will produce "attention weights" for the word vectors based on positional encodings and some other learnable parameters (key/query/value transforms). And then these can be "fed" to the decoder part of the network which will also consider word vectors that have been produced by the decoder so far and influence word selection by paying special attention to particular word combinations.

However LLMs clearly produce words in their output/response that do not occur anywhere inside the "prompt" text. So they must be using these "attention weights" to consider words from a wider vocabulary, which could be quite large.

Is it the case that the decoder "considers" each possible word in it's entire vocabulary when producing an output word? For example I'm imagining an input layer to a NN with several thousand nodes (one per word vector in dictionary) on the input then these are "combined" through some operation with attention weights (from the encoder and decoder "attention section") producing values for most word vectors that are very low (so below the threshold for some activation function) but each word is still "considered" to an extent? Or are only a subset of words considered in some way?

Answer 1

I guess this picture is not quite right: "And then these [attention weights?] can be 'fed' to the decoder part of the network which will also consider word vectors that have been produced by the decoder so far and influence word selection by paying special attention to particular word combinations."

The general picture I have learned goes like this:

1. One-hot word vectors are sent through the embedding matrix $W$ and yield lower-dimensional word embeddings.

2. In each decoder layer the word vectors are enriched by the attention mechanism, but only by contextual information from previous tokens ("masked attention").

3. So the first token doesn't get enriched at all, its information content remains the same.

4. The last token takes up the maximum of information available and after the last decoder represents the context as a whole.

5. It's only the enriched word embedding (= context embedding) of the last token that is now sent through the transpose $W^T$ of the word embedding matrix (the final decoder, different from the transformer decoders). It yields the logits vector.

Other models than GPT may use other final decoders than $W^T$.

The following diagram depicts this. The shades of gray of the word vectors passed through the transformer decoders display their informational content (or density).

Answer 2

After forwarding the prompt through the whole transformer architecture, the output is forwarded through one final linear layer.

"This final linear layer maps the output of the model back to the size of the 'vocabulary'. I.e. if you want to predict the next letter in a message, it would map to 26 (letters) + additional stuff (such as .,-!? etc.)."

For each possible output token, it predicts the probability (between 0 and 1) that that token is the next token. For models such as GPT-4, this last linear layer maps to a 10.000+ dimensional output (it includes arabian/chinese/japanese etc tokens).

Regarding your question:

Is it the case that the decoder "considers" each possible word in it's entire vocabulary when producing an output word?

Yes. Each output token is considered as a possible option.