3
$\begingroup$

Transformers are modified heavily in recent research. But what exactly makes a transformer a transformer? What is the core part of a transformer? Is it the self-attention, the parallelism, or something else?

$\endgroup$
1
  • 2
    $\begingroup$ When you say "Transformers are modified heavily in recent research", which research are you talking about that "modified heavily" the original transformer? In any case, here and here are 2 related questions. $\endgroup$
    – nbro
    May 27 '21 at 8:58
1
$\begingroup$

There is not one answer to this question, but one could argue that transformers heavily rely on

  • transforming each input into latent subspaces of queries, keys and values in order to generate attention score
  • a pool of transformations of the attention vectors (multi-head) according to which models can capture richer interpretations as different sections of the input embedding can attend different per-head subspaces that link back to each input
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.