If masked attention tells which token precedes which token, why are positional embeddings required?

Question

I'm learning about transformers and their components, and I understand that:

Masked attention ensures that each token can only attend to previous tokens (or itself) in causal language models, effectively encoding the order of tokens.

Positional embeddings are added to token embeddings to incorporate position information explicitly.

This makes me wonder: if masked attention already respects the sequence of tokens by enforcing causality, why do we still need positional embeddings? Doesn't the masking already capture the token order implicitly?

Can we get away without using the positional embedding if the masked attention already attends to the need of positioning of token ?

Answer 1

Without positional embeddings, the self-attention mechanism even with causal mask is permutation-invariant, meaning it simply treats the causally attended input sequence portion as a bag of words without inherent order which is necessary for many common NLP tasks where sequence matters such as understanding syntax or more high-level semantic temporal relations. For example, without positional encoding information in the embeddings, LLM would struggle to differentiate sequences between "the lion chased the tiger" and "the tiger chased the lion" both of which are outside the mask.

Of course you could use relative positional encoding or adding additional bias to model the same in the self-attention mechanism directly.