What information does the word embedding in Transformers will encode about the word when analysed outside of the model?

Question

Word2vec and similar architectures create word embedding vectors as a byproduct from a supervised learning task, where they need to predict the correct context word. Consequently, the inner representation of words inside this network will preserve some form of proximity-based word similarity based on the used corpus. When extracted, we can observe this via measuring cosine similarity between words, which will result in values close to 1 for words often occurring in each other's proximity and close to -1 for words that are highly infrequent together.

Thus, I would consider the word2vec embedding vectors to be quite interpretable regarding their meaning. What about transformers?

Transformers produce a similar inner representation of words, but than they alter them and recombine them through the attention mechanism multiple times, in order to solve the seq2seq learning task. What will the initial embedding before the first encoding really mean, if anything? Do they have any value when separated from the transformer? Like for example, the vectors generated in a word2vec model can be extracted and used in a downstream task. Is it reasonable to use the embedding vectors from a transformer for any downstream task?

Answer 1

That’s a very interesting question. From what I understand, transformer embedding vectors are not as interpretable as word2vec embedding vectors regarding their meaning. They are more dependent on the context and the attention mechanism that modifies them throughout the layers of the transformer. However, they can still be useful for some downstream tasks, such as text classification, sentiment analysis, or named entity recognition. They can also be used to generate text embeddings for the whole sequence by using the [CLS] token output vector or other strategies.