I am watching the video Attention Is All You Need by Yannic Kilcher.

My question is: what is the intuition behind the dot product attention?

$$A(q,K, V) = \sum_i\frac{e^{q.k_i}}{\sum_j e^{q.k_j}} v_i$$


$$A(Q,K, V) = \text{softmax}(QK^T)V$$


1 Answer 1


Let's start with a bit of notation and a couple of important clarifications.

$\mathbf{Q}$ refers to the query vectors matrix, $q_i$ being a single query vector associated with a single input word.

$\mathbf{V}$ refers to the values vectors matrix, $v_i$ being a single value vector associated with a single input word.

$\mathbf{K}$ refers to the keys vectors matrix, $k_i$ being a single key vector associated with a single input word.

Where do these matrices come from? Something that is not stressed out enough in a lot of tutorials is that these matrices are the result of a matrix product between the input embeddings and 3 matrices of trained weights: $\mathbf{W_q}$, $\mathbf{W_v}$, $\mathbf{W_k}$.

The fact that these three matrices are learned during training explains why the query, value and key vectors end up being different despite the identical input sequence of embeddings. It also explains why it makes sense to talk about multi-head attention. Performing multiple attention steps on the same sentence produces different results, because, for each attention 'head', new $\mathbf{W_q}$, $\mathbf{W_v}$, $\mathbf{W_k}$ are randomly initialised.

Another important aspect not stressed out enough is that for the encoder and decoder first attention layers, all the three matrices comes from the previous layer (either the input or the previous attention layer) but for the encoder/decoder attention layer, the $\mathbf{Q}$ matrix comes from the previous decoder layer, whereas the $\mathbf{V}$ and $\mathbf{K}$ matrices come from the encoder. And this is a crucial step to explain how the representation of two languages in an encoder is mixed together.

Once computed the three matrices, the transformer moves on to the calculation of the dot product between query and key vectors. The dot product is used to compute a sort of similarity score between the query and key vectors. Indeed, the authors used the names query, key and value to indicate that what they propose is similar to what is done in information retrieval. For example, in question answering, usually, given a query, you want to retrieve the closest sentence in meaning among all possible answers, and this is done by computing the similarity between sentences (question vs possible answers).

Of course, here, the situation is not exactly the same, but the guy who did the video you linked did a great job in explaining what happened during the attention computation (the two equations you wrote are exactly the same in vector and matrix notation and represent these passages):

  • closer query and key vectors will have higher dot products.
  • applying the softmax will normalise the dot product scores between 0 and 1.
  • multiplying the softmax results to the value vectors will push down close to zero all value vectors for words that had a low dot product score between query and key vector.

In the paper, the authors explain the attention mechanisms saying that the purpose is to determine which words of a sentence the transformer should focus on. I personally prefer to think of attention as a sort of coreference resolution step. The reason why I think so is the following image (taken from this presentation by the original authors).

enter image description here

This image shows basically the result of the attention computation (at a specific layer that they don't mention). Bigger lines connecting words mean bigger values in the dot product between the words query and key vectors, which means basically that only those words value vectors will pass for further processing to the next attention layer. But, please, note that some words are actually related even if not similar at all, for example, 'Law' and 'The' are not similar, they are simply related to each other in these specific sentences (that's why I like to think of attention as a coreference resolution). Computing similarities between embeddings would never provide information about this relationship in a sentence, the only reason why transformer learn these relationships is the presences of the trained matrices $\mathbf{W_q}$, $\mathbf{W_v}$, $\mathbf{W_k}$ (plus the presence of positional embeddings).

  • 1
    $\begingroup$ Ive been searching for how the attention is calculated, for the past 3 days. Your answer provided the closest explanation. Thank you. If you have more clarity on it, please write a blog post or create a Youtube video. It'd be a great help for everyone. $\endgroup$
    – Nav
    Jul 29, 2020 at 16:16
  • 3
    $\begingroup$ @Nav Hi, sorry but I saw your comment only now. I'm not really planning to write a blog post on this topic, mainly because I think that there are already good tutorials and video around that describe transformers in detail. Also, I saw that new posts are share every month, this one for example is really well made, hope you'll find it useful: peterbloem.nl/blog/transformers $\endgroup$ Aug 31, 2020 at 14:34
  • $\begingroup$ @Avatrin The weight matrices Eduardo is talking about here are not the raw dot product softmax wij that Bloem is writing about at the beginning of the article. The weight matrices here are an arbitrary choice of a linear operation that you make BEFORE applying the raw dot product self attention mechanism. These can technically come from anywhere, sure, but if you look at ANY implementation of the transformer architecture you will find that these are indeed learned parameters. Bloem covers this in entirety actually, so I don't quite understand your implication that Eduardo needs to reread it $\endgroup$ Mar 13, 2022 at 9:57
  • $\begingroup$ @TimSeguine Those linear layers are before the "scaled dot-product attention" as defined in Vaswani (seen in both equation 1 and figure 2 on page 4). They are however in the "multi-head attention". OPs question explicitly asks about equation 1. There are no weights in it. $\endgroup$
    – Avatrin
    Mar 13, 2022 at 11:47
  • $\begingroup$ @Avatrin Yes that's true, the attention function itself is matrix valued and parameter free(And I never disputed that fact), but your original comment is still false: "the three matrices W_q, W_k and W_v are not trained". Neither how they are defined here nor in the referenced blog post is that true. $\endgroup$ Mar 13, 2022 at 14:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .