# Is there a notion of location in Transformer architecture in subsequent self-attention layers?

Transformer architecture (without position embedding) is by the very construction equivariant to the permutation of tokens. Given query $$Q \in \mathbb{R}^{n \times d}$$ and keys $$K \in \mathbb{R}^{n \times d}$$ and some permutation matrix $$P \in \mathbb{R}^{n \times n}$$, one has: $$Q \rightarrow P Q, K \rightarrow P K$$ $$A = \text{softmax} \left(\frac{Q K^T}{\sqrt{d}} \right) \rightarrow \text{softmax} \left(\frac{R Q K^T R^T}{\sqrt{d}} \right) = R \ \text{softmax} \left(\frac{Q K^T}{\sqrt{d}} \right) R^T = R A R^T$$ Without the positional embedding (learned of fixed), that breaks permutation symmetry to translational there is no notion of location. And with the positional embedding one introduces a notion - token $$x$$ is on the $$k$$-th position.

However, I wonder, whether this notion makes sense after the first self-attention layer.

The operation of producing the output (multiplication of attention by the value) $$x_{out} = A V$$ outputs some weighted sum of all tokens in the sequence, the token at the $$k$$-th position now has information from all the sequence.

So, I wonder, whether the notion of position (absolute or relative) still makes sense in this case?

My guess is, that since Transformers involve skip connections, these transfer the notion of the location to the next layers. But this depends on the relative magnitude between the activation of the given self-attention layer and the skip-connections