2
$\begingroup$

It's often assumed in literature that BERT embeddings are contextual representations of the corresponding word. That is, if the 5th word is "cold", then the 5th BERT embedding is a representation of that word, using context to disambiguate the word (e.g. determine whether it's to do with the illness or temperature).

However, because of the self-attention encoder layers, this embedding can in theory incorporate information from any of the other words in the text. BERT is trained using masked language modelling (MLM), which would encourage each embedding to learn enough to predict the corresponding word. But why wouldn't it contain additional information from other words? In other words, is there any reason to believe the BERT embeddings for different words contain well-separated information?

$\endgroup$

1 Answer 1

1
$\begingroup$

While an input with $n$ tokens generates an output with $n$ vectors, there is a lot of cross mixing of information. One word, or even one word sense, may have many different representations. The embeddings are very sensitive to other words in the sentence, even when the sense of the word does not change. To measure how the embeddings change, we could use the standard deviation of the variations in the vector entries. For example, the difference between the word vectors for run in "Run fast" and "Run fast." (notice the period) is 0.24, which is only about half the difference among random uses of the word run.

The authors of the BERT paper explored the possibility of generating contextualized word embeddings using BERT. Table 7 in the BERT paper explains different ways they tried for extracting word representations that would work well in a named entity extraction task. They concluded that the weighted sum of the last four hidden layers provides good contextualized word representations. However, there are still many other choices to make when it comes to developing contextualized word embeddings, such as how to go from the wordpieces that BERT uses to a single word vector. If you're looking to learn more about BERT-related topics, I found the paper on BERTology to be a great starting point (section 4.1 discusses word embeddings).

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .