# What is the intuition behind position-encoding?

It is clear that word positions are essential for the meaning of a sentence, and so are essential when feeding a sentence (= sequence of words) as a matrix of word embedding vectors into a transformer. I also have understood roughly how positions are encoded, but what I did not understand in the very begining is why just creating a matrix consisting of a number of word embedding vectors (as columns) with the columns in the same order as the words in the sentence does not suffice. A matrix with permutated columns obviously would "mean" something different - and sometimes nothing at all - like a pixel matrix would change its "meaning" when we permutated some pixel columns. Is there an intuitive explanation why position encoding vectors have to be added to the word embedding vectors. Why and how would the position information (which is still present in the input matrix) get lost otherwise?

(I have learned in the meanwhile that transformers are in general not permutation invariant, but that there are transformers that are: set transformers.)

• Transformers are permutation invariant, the positional encoding added (by summing or concatenation) does not change that fact. can you rephrase your question? May 11, 2023 at 7:09
• @N.Kiefer: I was misled by Lee's paper on set transformers which seems to suggest that general (non-set) transformers were not permutation invariant. So if the truth is that general transformers are in general permutation invariant, this would perfectly answer my question. May 11, 2023 at 10:22
• @N.Kiefer: Convolutional networks in turn are really not permutation invariant, right? That's where the analogy between pixel matrices (as inputs to CNNs) and word-embedding matrices (as inputs to Transformers, which are fully connected) fails, right? May 11, 2023 at 10:24