What makes a transformer a transformer?

Question

Transformers are modified heavily in recent research. But what exactly makes a transformer a transformer? What is the core part of a transformer? Is it the self-attention, the parallelism, or something else?

Answer 1

It's about self-attention, a mechanism that targets parallelism among other goals (see 1706.03762.pdf - Why Self-Attention).

From What Is a Transformer Model? | NVIDIA Blogs:

How Transformers Got Their Name

Attention is so key to transformers the Google researchers almost used the term as the name for their 2017 model. Almost.

“Attention Net didn’t sound very exciting,” said Vaswani, who started working with neural nets in 2011.

Jakob Uszkoreit, a senior software engineer on the team, came up with the name Transformer.

“I argued we were transforming representations, but that was just playing semantics,” Vaswani said.

You'll see the same sentiment in the first paragraph of Transformer (machine learning model).

This is not to say that self-attention is the only the kind of attention mechanism employed by a Transformer model; see 1706.03762.pdf - 3.2.3 Applications of Attention in our Model. It's also not to say that adding an attention mechanism to your model is to make it a "Transformer" model; the innovation of this particular model was to go "all out" on attention mechanisms (you'd also have to get rid of recurrent and convolutional components).

Answer 2

There is not one answer to this question, but one could argue that transformers heavily rely on

• transforming each input into latent subspaces of queries, keys and values in order to generate attention score
• a pool of transformations of the attention vectors (multi-head) according to which models can capture richer interpretations as different sections of the input embedding can attend different per-head subspaces that link back to each input