0
$\begingroup$

In Transformers, we add the positional embedding with the word embedding and then process it. But, what if the sum of the embeddings become same for different words at different position of a sentence? How would the model differentiate it?

$\endgroup$

2 Answers 2

0
$\begingroup$

The positional embedding values in transformers are fixed from the start, they are calculated only once before the start of training (or stored/reused from years ago in memory).

So there is only very specific word embedding vector that can "confuse" the combination of word + position vector values.

Suppose embedding for a word is so coincidentally unfortunate as to fall into this category. Then the prediction accuracy of the transformer will decrease as a result of this unfortunate property. This causes readjustment of the word embedding (since position embeddings are fixed and unchangable).

Thus, training will naturally eliminate any existing ambiguity of word + position embeddings between different word vectors.

$\endgroup$
0
$\begingroup$

This would be incredibly difficult just due to the size of these embedding vectors and the fact that we are performing 2D rotations (many in parallel). The larger this embedding space the more the vectors will be spread out. So the main argument is not that it is impossible but that it is incredibly unlikely. The odds that any 2 vectors are a series of 2D rotations apart is low and becomes even lower in large embedding spaces.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .