3
$\begingroup$

I'm wondering what the origins of the transformer as proposed in Attention Is All You Need are. The paper itself provides some interesting pointers to the literature on self-attention such as:

  1. A Decomposable Attention Model for Natural Language Inference
  2. A STRUCTURED SELF-ATTENTIVE SENTENCE EMBEDDING
  3. NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE

It seems like using attentional mechanisms was widespread (at least in combination with recurrent networks) and 'A Decomposable Attention Model for Natural Language Inference' paper from 2016 already conjectured that scaling attention might be feasible. Is it from this prior work 'only' an engineering leap? Or what additional papers at the time likely influenced the architecture?

$\endgroup$

1 Answer 1

3
$\begingroup$

An influential predecessor paper is indeed the work on NEURAL MACHINE TRANSLATION BY JOINTLY LEARNING TO ALIGN AND TRANSLATE. The paper outlines an attentional mechanism that is similar to the computation of the actual attentional weights in the transformer paper. See the paper and picture below for details.

Model overview of attention mechanism

When the authors of the paper visualize their attention weights they obtain the following results. The inputs into the attention computation are not called queries and keys yet but are conceptually similar.

Attention weight visualization

See this video for a detailed explanation.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .