I know the original Transformer and the GPT (1-3) use two slightly different positional encoding techniques.
More specifically, in GPT they say positional encoding is learned. What does that mean? OpenAI's papers don't go into detail very much.
How do they really differ, mathematically speaking?
The purpose of introduction of positional encoding is to insert a notion of location of a given token in the sequence. Without it, due to the permutation equivariance (symmetry under the token permutation) there will be no notion of relative order inside a sequence.
Given a token at $\text{pos}$-th position we would like to make the model understand, that this token is at particular position. See pretty nice blog here - https://kazemnejad.com/blog/transformer_architecture_positional_encoding/.
Fixed encoding
In the original Transformer one uses a fixed map from the token position $i$ to the embedding vector added to the original embedding: $$ \begin{aligned} PE(\text{pos}, 2i) &= \sin(\text{pos} / 10000^{2i / d_{\text{model}}}) \\ PE(\text{pos}, 2i + 1) &= \cos(\text{pos} / 10000^{2i / d_{\text{model}}}) \end{aligned} $$
Here $\text{pos}$ is an index of the token in sequence, and $2i, 2i+1$ correspond to the dimension inside the embedding.
Learned encoding
Another strategy is to make map for $\text{pos}$ to the embedding vector of dimension $d_{\text{model}}$ learnable. One initializes somehow for each position in the sequence vector of positional embedding for each position from $0$ to $\text{max_length}$ and during the training these vectors are updated by gradient descent.
As far as I understood, the difference is the following: original Transformers use a fixed type of encoding, based on sine/cosine functions.
On the other hand, GPT produces two embedding vectors: one of the input tokens, as usual in language models, and another for token positions themselves.
