I know the original Transformer and the GPT (1-3) use two slightly different positional encoding techniques.

More specifically, in GPT they say positional encoding is learned. What does that mean? OpenAI's papers don't go into detail very much.

How do they really differ, mathematically speaking?


As far as I understood, the difference is the following: original Transformers use a fixed type of encoding, based on sine/cosine functions.

GPT on the other hand, produces two embedding vectors: one of the input tokens, as usual in language models, and another for token positions themselves!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.