My understanding is that GPT uses the same embedding matrix for both inputs and output: Let $V$ be the vocab size, $D$ the number of embedding dimensions, and $E$ be a $V \times D$ embedding matrix:
- On input, if $x$ is a one-hot $V$-dimensional vector, GPT uses $Ei$.
- On output, if $\hat y$ is a $D$-dimensional prediction vector, GPT uses softmax($E^\top{\hat y}$) as its predictions.
Q1. Is the above correct?
I cannot find this stated clearly in the paper, but it is stated explicitly here. It's also clearly implied by the parameter count listed here, and argued for as best practice here. Yet, for example, Karpathy's mini-GPT implementation seems to use two different matrices:
self.tok_emb = nn.Embedding(config.vocab_size, config.n_embd) # <--- This would be E
self.pos_emb = nn.Parameter(torch.zeros(1, config.block_size, config.n_embd))
self.drop = nn.Dropout(config.embd_pdrop)
# transformer
self.blocks = nn.Sequential(*[Block(config) for _ in range(config.n_layer)])
# decoder head
self.ln_f = nn.LayerNorm(config.n_embd)
self.head = nn.Linear(config.n_embd, config.vocab_size, bias=False) # <--- This has the same dimensions as Etranspose but is clearly a different matrix
Q2. If it is correct, how does can it work?
This seems to be tasking $E$ with two very different, even opposing, functions:
- Map vocab to their meaning on the input side; higher magnitude indicates "more meaning"
- Map meaning to the most likely vocab on the output side; higher magnitude indicates greater likelihood
When outputting, we want the softmax to be highest when the word is most likely; magnitude of the output matrix should be roughly proportional to how likely the word is two appear.
Yet, when inputting, magnitude has nothing to do with likelihood. Magnitude on the input side captures some element of meaning: perhaps how extreme or intense the meaning is, perhaps another aspect (not necessarily easily interpreted).
