1
$\begingroup$

Transformers are modified heavily in recent research. But what exactly makes a transformer a transformer? What is the core part of a transformer? Is it the self-attention or the parallelism or something else?

$\endgroup$
1
  • 2
    $\begingroup$ Hello. When you say "Transformers are modified heavily in recent research", which research are you talking about that "modified heavily" the original transformer? $\endgroup$
    – nbro
    May 27 at 8:58
1
$\begingroup$

There is not one answer to this question, but one could argue that transformers heavily rely on

  • transforming each input into latent subspaces of queries, keys and values in order to generate attention score
  • a pool of transformations of the attention vectors (multi-head) according to which models can capture richer interpretations as different sections of the input embedding can attend different per-head subspaces that link back to each input
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.