Transformers are modified heavily in recent research. But what exactly makes a transformer a transformer? What is the core part of a transformer? Is it the self-attention or the parallelism or something else?
There is not one answer to this question, but one could argue that transformers heavily rely on
- transforming each input into latent subspaces of queries, keys and values in order to generate attention score
- a pool of transformations of the attention vectors (multi-head) according to which models can capture richer interpretations as different sections of the input embedding can attend different per-head subspaces that link back to each input