In Word2Vec, the embeddings don't depend on the context.
But in Transformers, the embeddings depend on the context.
So how are the words' embeddings set at inference time?
How do Transformers compute the words embeddings at inference time since the embeddings are dynamic?
1
-
$\begingroup$ Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. $\endgroup$– Community BotOct 17, 2022 at 14:39
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
3
Transformers do still use pre-trained embeddings but these are then given context by the positional encoding and self-attention blocks of the architecture.
I will explain in the context of a sequence-to-sequence transformer used for translating a sentence from one language to another (e.g. that used in Attention is All You Need):
At inference time the input sequence process (i.e. the encoder) is the same as during training time: each word in the input sequence is mapped to the embedding vector for that word, and 'given context' by the positional encoding and encoder self-attention blocks.
In the decoder, at inference time we are predicting the translation of each word in the input sequence, one at a time. So instead of feeding in the target translated sentence to the decoder, we feed in an empty sequence with a start-of-sentence token. In much the same way as in the encoder, each token in this sentence is given context by the positional encoding and self-attention blocks. The next word in the output sequence is then predicted by the rest of the decoder layers and this is appended to the output sentence. This output sentence (now consisting of the start-of-sentence token and the first word(s)) is then fed into the decoder to predict the next word, until the end-of-sentence token is predicted.
So in answer to your question, the blocks of the transformer that give the word embeddings context are the same during training and inference.
-
$\begingroup$ So to sum up, the word embeddings are the same as in training time, the positional encoding differs, that's right? $\endgroup$– JulienNov 3, 2022 at 13:06
-
$\begingroup$ Yes - the word embeddings are the same in training and inference. The word embedding stage is just a lookup from word to pre-trained embedding representation for that word. As you say, these are then transformed by the positional encoding (depending on where the word is in the sentence) and by the attention layers (i.e. weights of the key, value and query matrices that were learnt during training). $\endgroup$– AtticusNov 3, 2022 at 15:20
-
$\begingroup$ it might also help to point out that that self-attention isn't technically an embedding and isn't learned but computed on the fly for each input (whether at training or at inference). $\endgroup$– waxalasNov 3, 2022 at 18:09
$\begingroup$
$\endgroup$
Embedding doesn't provide context like self-attention does. Embedding is what you do to represent your training data (words or pictures) into numerical vectors.
Word embedding is learned and will be the same at training and inference. For example, "Transformers are cool" will be embedded as [122, 5, 43], while "How are you" will be embedded as [10, 5, 2].
Positional embedding is a deterministic function. It is not only the same at training and inference, but it is also the same on every sample in your training data. Both "Transformers are cool" and "How are you" will have positional embedding [1, 2, 3]. (Positional embedding could also be a learned transformation, but in the original paper, they didn't find much improvement in using one).
Now, self-attention is what gives inputs context (for example, "are" and "you" are related and conjugated accordingly in the sentence "How are you"). Self-attention is a matrix of weights which is indeed dynamic since it is computed on the fly during training. But you can also compute self-attention on your new predictor on the fly during inference.
