What is the intuition behind the dot product attention?

Question

I am watching the video Attention Is All You Need by Yannic Kilcher.

My question is: what is the intuition behind the dot product attention?

$$A(q,K, V) = \sum_i\frac{e^{q.k_i}}{\sum_j e^{q.k_j}} v_i$$

becomes:

$$A(Q,K, V) = \text{softmax}(QK^T)V$$

Answer 1

Let's start with a bit of notation and a couple of important clarifications.

$\mathbf{Q}$ refers to the query vectors matrix, $q_i$ being a single query vector associated with a single input word.

$\mathbf{V}$ refers to the values vectors matrix, $v_i$ being a single value vector associated with a single input word.

$\mathbf{K}$ refers to the keys vectors matrix, $k_i$ being a single key vector associated with a single input word.

Where do these matrices come from? Something that is not stressed out enough in a lot of tutorials is that these matrices are the result of a matrix product between the input embeddings and 3 matrices of trained weights: $\mathbf{W_q}$, $\mathbf{W_v}$, $\mathbf{W_k}$.

The fact that these three matrices are learned during training explains why the query, value and key vectors end up being different despite the identical input sequence of embeddings. It also explains why it makes sense to talk about multi-head attention. Performing multiple attention steps on the same sentence produces different results, because, for each attention 'head', new $\mathbf{W_q}$, $\mathbf{W_v}$, $\mathbf{W_k}$ are randomly initialised.

Another important aspect not stressed out enough is that for the encoder and decoder first attention layers, all the three matrices comes from the previous layer (either the input or the previous attention layer) but for the encoder/decoder attention layer, the $\mathbf{Q}$ matrix comes from the previous decoder layer, whereas the $\mathbf{V}$ and $\mathbf{K}$ matrices come from the encoder. And this is a crucial step to explain how the representation of two languages in an encoder is mixed together.

Once computed the three matrices, the transformer moves on to the calculation of the dot product between query and key vectors. The dot product is used to compute a sort of similarity score between the query and key vectors. Indeed, the authors used the names query, key and value to indicate that what they propose is similar to what is done in information retrieval. For example, in question answering, usually, given a query, you want to retrieve the closest sentence in meaning among all possible answers, and this is done by computing the similarity between sentences (question vs possible answers).

Of course, here, the situation is not exactly the same, but the guy who did the video you linked did a great job in explaining what happened during the attention computation (the two equations you wrote are exactly the same in vector and matrix notation and represent these passages):

• closer query and key vectors will have higher dot products.
• applying the softmax will normalise the dot product scores between 0 and 1.
• multiplying the softmax results to the value vectors will push down close to zero all value vectors for words that had a low dot product score between query and key vector.

In the paper, the authors explain the attention mechanisms saying that the purpose is to determine which words of a sentence the transformer should focus on. I personally prefer to think of attention as a sort of coreference resolution step. The reason why I think so is the following image (taken from this presentation by the original authors).

This image shows basically the result of the attention computation (at a specific layer that they don't mention). Bigger lines connecting words mean bigger values in the dot product between the words query and key vectors, which means basically that only those words value vectors will pass for further processing to the next attention layer. But, please, note that some words are actually related even if not similar at all, for example, 'Law' and 'The' are not similar, they are simply related to each other in these specific sentences (that's why I like to think of attention as a coreference resolution). Computing similarities between embeddings would never provide information about this relationship in a sentence, the only reason why transformer learn these relationships is the presences of the trained matrices $\mathbf{W_q}$, $\mathbf{W_v}$, $\mathbf{W_k}$ (plus the presence of positional embeddings).