Word2vec and similar architectures create word embedding vectors as a byproduct from a supervised learning task, where they need to predict the correct context word. Consequently, the inner representation of words inside this network will preserve some form of proximity-based word similarity based on the used corpus. When extracted, we can observe this via measuring cosine similarity between words, which will result in values close to 1 for words often occurring in each other's proximity and close to -1 for words that are highly infrequent together.
Thus, I would consider the word2vec embedding vectors to be quite interpretable regarding their meaning. What about transformers?
Transformers produce a similar inner representation of words, but than they alter them and recombine them through the attention mechanism multiple times, in order to solve the seq2seq learning task. What will the initial embedding before the first encoding really mean, if anything? Do they have any value when separated from the transformer? Like for example, the vectors generated in a word2vec model can be extracted and used in a downstream task. Is it reasonable to use the embedding vectors from a transformer for any downstream task?