0
$\begingroup$

In Transformer models, token embeddings are combined with positional encodings through element-wise addition to incorporate positional information. However, this raises a concern about the potential for different tokens in different positions to end up with identical embeddings.

For example, consider the following:

  • Embedding of token $A$: $[0.5, 0]$
  • Positional encoding at position $1$: $[1, 0]$
  • Combined representation for token $A$ at position $1$: $[0.5 + 1, 0 + 0] = [1.5, 0]$
  • Embedding of token $B$: $[1, 0]$
  • Positional encoding at position $2$: $[0.5, 0]$
  • Combined representation for token $B$ at position $2$: $[1 + 0.5, 0 + 0] = [1.5, 0]$

In this case, the combined representations for different tokens at different positions are the same.

Given that positional encodings are added element-wise to token embeddings, how do Transformer models ensure the uniqueness of token representations to prevent different token-position combinations from resulting in the same embedding vector? Specifically, how do they handle cases where different embeddings and positional encodings could potentially sum to the same vector?

$\endgroup$

2 Answers 2

0
$\begingroup$

Positional Encoding (PE) is a broad term encompassing various techniques meant to embed positional information within a matrix $x \in \mathbb{R}^{S \times d_{\text{model}}}$, where $S$ is the number of tokens, and $d_{\text{model}}$ is the size of each token. Assuming you are referring to the PE defined in "Attention is all you need" (Vaswani et al., 2017), then you have: $$\operatorname{PE}(p,2i) = \sin\left( \frac{p}{10,000^{\frac{2i}{d_{\text{model}}}}}\right)$$ $$\operatorname{PE}(p,2i+1) = \cos\left( \frac{p}{10,000^{\frac{2i}{d_{\text{model}}}}}\right)$$ where $d_{\text{model}}$ is the size of the embedding vector, $p$ is the position of the (token) embedding in the sequence, and $i$ is the feature index in the embedding space. As you can see, $\operatorname{PE}$ is not a function of the embeddings, but rather a determinsitc function of the position $p$ and the feature index $i$. This means that PE is the same across all input sequences.

This is also the reason why in many practical implementation, like here, PE is computed only once. Specifically, the PEs are precomputed for a fixed maximum sequence length and stored in a tensor. When an input sequence $x$ is processed, the corresponding PEs are sliced from this precomputed tensor (to match the dimensions) and added to $x$.

Therefore, back to your example, if your sequence is $$ \begin{bmatrix} A \\ B\\ \end{bmatrix} = \begin{bmatrix} 0.5 & 0.0 \\ 1.0 & 0.0 \\ \end{bmatrix} $$ you have $d_{\text{model}} = 2$ and $p, i \in \{0, 1\}$. First, compute the PEs: \begin{align} \operatorname{PE}(0,0) &= \sin\left( \frac{0}{10,000^{\frac{0}{2}}} \right) = \sin\left( 0 \right) = 0 \\ \operatorname{PE}(0,1) &= \cos\left( \frac{0}{10,000^{\frac{0}{2}}} \right) = \cos\left( 0 \right) = 1 \\ \operatorname{PE}(1,0) &= \sin\left( \frac{1}{10,000^{\frac{0}{2}}} \right) = \sin(1) \approx 0.8415 \\ \operatorname{PE}(1,1) &= \cos\left( \frac{1}{10,000^{\frac{0}{2}}} \right) = \cos(1) \approx 0.5403 \end{align}

This results in the PE tensor: $$ \operatorname{PE} = \begin{bmatrix} 0.0000 & 1.0000\\ 0.8415 & 0.5403\\ \end{bmatrix} $$ which will be the same for all input sequences. Your input sequence is then transformed as follows: $$ x' = x + \operatorname{PE} = \begin{bmatrix} 0.5 & 0.0 \\ 1.0 & 0.0 \\ \end{bmatrix} + \begin{bmatrix} 0.0000 & 1.0000\\ 0.8415 & 0.5403\\ \end{bmatrix} = \begin{bmatrix} 0.5000 & 1.0000\\ 1.8415 & 0.5403\\ \end{bmatrix} $$

Notably, each PE is unique for each position within an input sequence. Consequently, if two distinct tokens within the same input sequence, denoted as $x_p$ and $x_q$, yield identical embeddings after the application of their respective PEs: $$ x_p + \operatorname{PE}(p) = x_q + \operatorname{PE}(q)$$ several implications arise. First, the occurrence of such an event is highly improbable, given that $\operatorname{PE}(p) \neq \operatorname{PE}(q)$. Second, even if this rare event occurs, the model is capable of distinguishing between the tokens based on their positional contexts. Since $\operatorname{PE}(p) \neq \operatorname{PE}(q)$, then also $x_p \neq x_q$ (the embeddings are different). Thus, the information encoded by $x_p$ and $x_q$ are different, even though the final embeddings are the same.

$\endgroup$
8
  • $\begingroup$ What is meant by "$p$ is the position of the (token) embedding in the sequence"? "Position in the sequence" in what sense? The paper itself seems to be similarly vague about this. $\endgroup$ Commented Jul 10 at 15:59
  • 1
    $\begingroup$ Given the input sequence $[a, b, c, d]$, $p=0$ for $a$, $p=1$ for $b$, $p=2$ for $c$, $p=3$ for $d$. Here, $a$, $b$, $c$, $d$ are your tokens, hence each is a vector of size $d_{\text{model}}$. Transformers expect a sequence of tokens as input, not one token at a time. $\endgroup$
    – frad
    Commented Jul 10 at 16:04
  • $\begingroup$ If the final embeddings are the same, then how does the token embeddings $x_p$ and $x_q$ being different matter? Doesn’t the model use the final embedding state, not the intermediate token embedding states? Aren’t these intermediate token embedding states just a means to an end (the final embedding states)? $\endgroup$ Commented Jul 10 at 18:20
  • 1
    $\begingroup$ I am not sure I understand your point. If positional information is embedded within the initial embeddings, then the intermediate embeddings will also inherently contain that information, as they are derived from these initial embeddings. This is a mathematical matter: you have two vectors, $x_p$ and $x_q$ occurring in two different positions, $p$ and $q$. This implies $p \neq q \Rightarrow \operatorname{PE}(p) \neq \operatorname{PE}(q)$ by definition of PE. [...] $\endgroup$
    – frad
    Commented Jul 11 at 9:06
  • 1
    $\begingroup$ [...] By integrating the positional information into the token embeddings, we obtain $x_p + \operatorname{PE}(p)$ for the token at position $p$ and $x_q + \operatorname{PE}(q)$ for the token at position $q$. Given $\operatorname{PE}(p) \neq \operatorname{PE}(q)$ by definition, the equality $x_p + \operatorname{PE}(p) = x_q + \operatorname{PE}(q)$ holds iff $x_p \neq x_q$. This proves that in order to obtain the same representation for two tokens in two different positions, they must be different, thus they encode different information. $\endgroup$
    – frad
    Commented Jul 11 at 9:06
0
$\begingroup$

Long story short, they don't

But probabilistically speaking, given that they are supposed to be in $\mathbb{R}^n$, there is a probability 0 that they actually coincide

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .