It is clear that word positions are essential for the meaning of a sentence, and so are essential when feeding a sentence (= sequence of words) as a matrix of word embedding vectors into a transformer. I also have understood roughly how positions are encoded, but what I did not understand in the very begining is why just creating a matrix consisting of a number of word embedding vectors (as columns) with the columns in the same order as the words in the sentence does not suffice. A matrix with permutated columns obviously would "mean" something different - and sometimes nothing at all - like a pixel matrix would change its "meaning" when we permutated some pixel columns. Is there an intuitive explanation why position encoding vectors have to be added to the word embedding vectors. Why and how would the position information (which is still present in the input matrix) get lost otherwise?

(I have learned in the meanwhile that transformers are in general not permutation invariant, but that there are transformers that are: set transformers.)

  • $\begingroup$ Transformers are permutation invariant, the positional encoding added (by summing or concatenation) does not change that fact. can you rephrase your question? $\endgroup$
    – N. Kiefer
    May 11, 2023 at 7:09
  • 1
    $\begingroup$ @N.Kiefer: I was misled by Lee's paper on set transformers which seems to suggest that general (non-set) transformers were not permutation invariant. So if the truth is that general transformers are in general permutation invariant, this would perfectly answer my question. $\endgroup$ May 11, 2023 at 10:22
  • $\begingroup$ @N.Kiefer: Convolutional networks in turn are really not permutation invariant, right? That's where the analogy between pixel matrices (as inputs to CNNs) and word-embedding matrices (as inputs to Transformers, which are fully connected) fails, right? $\endgroup$ May 11, 2023 at 10:24

1 Answer 1


I understand the confusion. Although transformers are autoregressive (they predict something based on past information), they are not recurrent (do not have hidden states).

In fact, think of a transformer as a multi-layer perceptron. Each input neuron is fully connected to every neuron in the first hidden layer. If you would swap the places of the first and the second input neuron (including their associated weights), all the connections/weights stay the same, but the order of the neurons is different.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .