Is there a notion of location in Transformer architecture in subsequent self-attention layers?

Question

Transformer architecture (without position embedding) is by the very construction equivariant to the permutation of tokens. Given query $Q \in \mathbb{R}^{n \times d}$ and keys $K \in \mathbb{R}^{n \times d}$ and some permutation matrix $P \in \mathbb{R}^{n \times n}$, one has: $$ Q \rightarrow P Q, K \rightarrow P K $$ $$\begin{align} A &= \text{softmax} \left(\frac{Q K^T}{\sqrt{d}} \right) \\ &\rightarrow \text{softmax} \left(\frac{R Q K^T R^T}{\sqrt{d}} \right) \\ &= R \ \text{softmax} \left(\frac{Q K^T}{\sqrt{d}} \right) R^T \\ &= R A R^T \end{align}$$ Without the positional embedding (learned of fixed), that breaks permutation symmetry to translational there is no notion of location. And with the positional embedding one introduces a notion - token $x$ is on the $k$-th position.

However, I wonder, whether this notion makes sense after the first self-attention layer.

The operation of producing the output (multiplication of attention by the value) $$ x_{out} = A V $$ outputs some weighted sum of all tokens in the sequence, the token at the $k$-th position now has information from all the sequence.

So, I wonder, whether the notion of position (absolute or relative) still makes sense in this case?

My guess is, that since Transformers involve skip connections, these transfer the notion of the location to the next layers. But this depends on the relative magnitude between the activation of the given self-attention layer and the skip-connections

Answer 1

Note that the architecture of the encoder (and decoder) uses residual connections that add the original input to the output of the self attention layer. Thus, the information from the positional information will be preserved in the deeper layers.

The encoder also uses a position-wise mlp after the self-attention, after which we also apply the residual connection. For more details (and figures) on the encoder architecture see here.

Answer 2

Attention mechanism solves this problem by allowing the decoder to “look-back” at the encoder's hidden states based on its current state. This allows the decoder to extract only relevant information about the input tokens at each decoding, thus learning more complicated dependencies between the input and the output.