In Word2Vec, the embeddings don't depend on the context.
But in Transformers, the embeddings depend on the context.
So how are the words' embeddings set at inference time?
Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.Sign up to join this community
Transformers do still use pre-trained embeddings but these are then given context by the positional encoding and self-attention blocks of the architecture.
I will explain in the context of a sequence-to-sequence transformer used for translating a sentence from one language to another (e.g. that used in Attention is All You Need):
At inference time the input sequence process (i.e. the encoder) is the same as during training time: each word in the input sequence is mapped to the embedding vector for that word, and 'given context' by the positional encoding and encoder self-attention blocks.
In the decoder, at inference time we are predicting the translation of each word in the input sequence, one at a time. So instead of feeding in the target translated sentence to the decoder, we feed in an empty sequence with a start-of-sentence token. In much the same way as in the encoder, each token in this sentence is given context by the positional encoding and self-attention blocks. The next word in the output sequence is then predicted by the rest of the decoder layers and this is appended to the output sentence. This output sentence (now consisting of the start-of-sentence token and the first word(s)) is then fed into the decoder to predict the next word, until the end-of-sentence token is predicted.
So in answer to your question, the blocks of the transformer that give the word embeddings context are the same during training and inference.
Embedding doesn't provide context like self-attention does. Embedding is what you do to represent your training data (words or pictures) into numerical vectors.
Word embedding is learned and will be the same at training and inference. For example, "Transformers are cool" will be embedded as [122, 5, 43], while "How are you" will be embedded as [10, 5, 2].
Positional embedding is a deterministic function. It is not only the same at training and inference, but it is also the same on every sample in your training data. Both "Transformers are cool" and "How are you" will have positional embedding [1, 2, 3]. (Positional embedding could also be a learned transformation, but in the original paper, they didn't find much improvement in using one).
Now, self-attention is what gives inputs context (for example, "are" and "you" are related and conjugated accordingly in the sentence "How are you"). Self-attention is a matrix of weights which is indeed dynamic since it is computed on the fly during training. But you can also compute self-attention on your new predictor on the fly during inference.