Transformer architecture (without position embedding) is by the very construction equivariant to the permutation of tokens. Given query $Q \in \mathbb{R}^{n \times d}$ and keys $K \in \mathbb{R}^{n \times d}$ and some permutation matrix $P \in \mathbb{R}^{n \times n}$, one has: $$ Q \rightarrow P Q, K \rightarrow P K $$ $$ A = \text{softmax} \left(\frac{Q K^T}{\sqrt{d}} \right) \rightarrow \text{softmax} \left(\frac{R Q K^T R^T}{\sqrt{d}} \right) = R \ \text{softmax} \left(\frac{Q K^T}{\sqrt{d}} \right) R^T = R A R^T $$ Without the positional embedding (learned of fixed), that breaks permutation symmetry to translational there is no notion of location. And with the positional embedding one introduces a notion - token $x$ is on the $k$-th position.

However, I wonder, whether this notion makes sense after the first self-attention layer.

The operation of producing the output (multiplication of attention by the value) $$ x_{out} = A V $$ outputs some weighted sum of all tokens in the sequence, the token at the $k$-th position now has information from all the sequence.

So, I wonder, whether the notion of position (absolute or relative) still makes sense in this case?

enter image description here

My guess is, that since Transformers involve skip connections, these transfer the notion of the location to the next layers. But this depends on the relative magnitude between the activation of the given self-attention layer and the skip-connections


Attention mechanism solves this problem by allowing the decoder to “look-back” at the encoder's hidden states based on its current state. This allows the decoder to extract only relevant information about the input tokens at each decoding, thus learning more complicated dependencies between the input and the output.

  • $\begingroup$ But what about fully encoder architectures? Like ViTs in computer vision? $\endgroup$ Oct 31 '21 at 12:31
  • $\begingroup$ In addition, as far as I remember, usually, decoder receives information from the last layer of encoder. There are multiple of them, hence the question remains. $\endgroup$ Oct 31 '21 at 13:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.