# What is the intuition behind the dot product attention?

I am watching Attention all you need, In that what is the intuition behind the dot product attention?

$$A(q,K, V) = \sum_i\frac{e^{q.k_i}}{\sum_j e^{q.k_j}} v_i$$

becomes:

$$A(Q,K, V) = softmax(QK^T)V$$

• Maybe you can describe what those vectors represent. What are $e$, $v$, $q$ and $A$ supposed to represent in your context? Anyway, I am not familiar with the details of that paper and video, but the dot product can be seen as a similarity measure between vectors. In fact, many similarity measures are just applications of the dot product. See e.g. math.stackexchange.com/q/689022/168764. Maybe later I may provide an answer to your question, if I decide to investigate that paper and video.
– nbro
Apr 11 '20 at 13:26

Let's start with a bit of notation and a couple of important clarification.

Q refers to the query vectors matrix, $$q_i$$ being a single query vector associated with a single input word.

V refers to the values vectors matrix, $$v_i$$ being a single value vector associated with a single input word.

K refers to the keys vectors matrix, $$k_i$$ being a single key vector associated with a single input word.

Where do these matrices come from? Something that is not stressed out enough in lot of tutorials is that these matrices are the result of a matrix product between the input embeddings and 3 matrices of trained weights: W$$_{\textbf{q}}$$, W$$_{\textbf{v}}$$, W$$_{\textbf{k}}$$.

The fact that these three matrices are learned during training explain why the query, value and key vectors end up being different despite the identical input sequence of embeddings. It also explain why it make sense to talk about multi-head attention. Performing multiple attention steps on the same sentence produce different results because for each attention 'head' new W$$_{\textbf{q}}$$, W$$_{\textbf{v}}$$, W$$_{\textbf{k}}$$ are randomly initialised.

Another important aspect not stressed out enough is that for the encoder and decoder first attention layers, all the three matrices comes from the previous layer (either the input or the previous attention layer) but for the encoder/decoder attention layer, the Q matrix comes from the previous decoder layer, whereas the V and K matrices come from the encoder. And this is a crucial step to explain how the representation of two languages in an encoder are mixed together.

Once computed the three matrices, the transformer move on on the calculation of the dot product between query and key vectors. As nbro wrote in its comment, the dot product is used to compute a sort of similarity score between the query and key vectors. Indeed, the authors used the names query, key and value to indicate that what they propose is similar to what is done in information retrieval. For example in question answering usually given a query you want to retrive the closer sentence in meaning among all possible answers, and this is done by computing the similarity between sentences (question vs possible answers).

Of course here the situation is not exactly the same, but the guy who did the video you linked did a great job in explaining what happened during the attention computation (the two equations you wrote are exactly the same in vector and matrix notation and represent these passages):

• closer query and key vectors will have higher dot products.
• applying the softmax will normalise the dot product scores between 0 and 1.
• multiplying the softmax results to the value vectors will push down close to zero all value vectors for words which had a low dot product score between query and key vector.

In the paper the authors explain the attention mechanisms saying that the purpose is to determine which words of a sentence the transformer should focus on. I personally prefer to think at attention as a sort of coreference resolution step, the reason why I think so is the following image (taken from this presentation by the original authors).

This image shows basically the result of the attention computation (at a specific layer that they don't mention). Bigger lines connecting words mean bigger values in the dot product between the words query and key vectors, which means basically that only those words value vectors will pass for further processing to the next attention layer. But please note that some words are actually related even if not similar at all, for example 'Law' and 'The' are not similar, they are simply related to each other in this specific sentences (that's why I like to think to attention as coreference resolution). Computing similarities between embeddings would never provide information about this relation in a sentence, the only reason why transformer learn these relationship is the presences of the trained matrices W$$_{\textbf{q}}$$, W$$_{\textbf{v}}$$, W$$_{\textbf{k}}$$ (plus the presence of positional embeddings).

• Ive been searching for how the attention is calculated, for the past 3 days. Your answer provided the closest explanation. Thank you. If you have more clarity on it, please write a blog post or create a Youtube video. It'd be a great help for everyone.
– Nav
Jul 29 '20 at 16:16
• @Nav Hi, sorry but I saw your comment only now. I'm not really planning to write a blog post on this topic, mainly because I think that there are already good tutorials and video around that describe transformers in detail. Also, I saw that new posts are share every month, this one for example is really well made, hope you'll find it useful: peterbloem.nl/blog/transformers Aug 31 '20 at 14:34
• Great explanation!!! Mar 11 at 5:00