# How does GPT use the same embedding matrix for both input and output?

My understanding is that GPT uses the same embedding matrix for both inputs and output: Let $$V$$ be the vocab size, $$D$$ the number of embedding dimensions, and $$E$$ be a $$V \times D$$ embedding matrix:

• On input, if $$x$$ is a one-hot $$V$$-dimensional vector, GPT uses $$Ei$$.
• On output, if $$\hat y$$ is a $$D$$-dimensional prediction vector, GPT uses softmax($$E^\top{\hat y}$$) as its predictions.

## Q1. Is the above correct?

I cannot find this stated clearly in the paper, but it is stated explicitly here. It's also clearly implied by the parameter count listed here, and argued for as best practice here. Yet, for example, Karpathy's mini-GPT implementation seems to use two different matrices:

self.tok_emb = nn.Embedding(config.vocab_size, config.n_embd) # <--- This would be E
self.pos_emb = nn.Parameter(torch.zeros(1, config.block_size, config.n_embd))
self.drop = nn.Dropout(config.embd_pdrop)
# transformer
self.blocks = nn.Sequential(*[Block(config) for _ in range(config.n_layer)])
self.ln_f = nn.LayerNorm(config.n_embd)
self.head = nn.Linear(config.n_embd, config.vocab_size, bias=False) # <--- This has the same dimensions as Etranspose but is clearly a different matrix



## Q2. If it is correct, how does can it work?

This seems to be tasking $$E$$ with two very different, even opposing, functions:

• Map vocab to their meaning on the input side; higher magnitude indicates "more meaning"
• Map meaning to the most likely vocab on the output side; higher magnitude indicates greater likelihood

When outputting, we want the softmax to be highest when the word is most likely; magnitude of the output matrix should be roughly proportional to how likely the word is two appear.

Yet, when inputting, magnitude has nothing to do with likelihood. Magnitude on the input side captures some element of meaning: perhaps how extreme or intense the meaning is, perhaps another aspect (not necessarily easily interpreted).

A GPT produces output based on its own previous output, so it must be able to understand its output.

The learning input is provided as a stream of tokens, and these tokens are defined before learning starts. So it has to use the same set of tokens to understand its own output. The set of possible output tokens is fixed, it learns only to assign probabilities of the next token.

Looking on it in a per token perspective, when it gets a token as input, it learns that it has a non-zero probability to be following the previous input. If the same happens again, it learned that the probability is higher than previously assumed. The probability is expressed as a number of occurrences in the input, and a number per token in the output, but both are about the relative probability of tokens following the previous text.

Yes, GPT uses the same embedding matrix. See here. Regarding your second question - on the input side, a given token selects one row from the token embedding matrix; it is not clear that higher magnitude signifies anything. On the output side, the magnitude of the output vector $$h_n$$ also doesn't signify anything. However, once you take the dot product of the output vector with each of the previous "rows" of the token embedding matrix, this gives you a dot-product similarity of the computed output with the pre-stored embeddings. A large value for a given token element means the model is more likely to predict that token.

• Please don't upload text as images, it isn't searchable. Use MathJax whenever possible. Thanks.
– Rob
Aug 21, 2022 at 15:04