I'm trying to understand the difference between the concept of self-attention and multi-head attention. The latter is not actually too clear to me.

I understand that, in the case of self-attention, we start with a feature matrix $X \in \mathbb{R}^{n \times d}$, and then we use the same linear transformation $W$ to produce

\begin{align} Q &= XW \\ K &= XW \\ V &= XW \end{align}

and then we compute the following

$$X' = \text{softmax} \left(\frac{Q\cdot K^T}{\sqrt{d}} \right)V$$

where $X' \in \mathbb{R}^{n \times d}$ is a new version of the input matrix, where the pairwise interactions between the points will be encoded.

What is multi-head attention doing, from a mathematical point of view, and what's the difference? I know we are going to use three different linear transformations in this case (so no weight-sharing), but what exactly will be encoded using three different $W$? Maybe it's more the conceptual view that it's not too clear in this case.


1 Answer 1


First of all, I believe that in self-attention mechanism for Query, Key and Value vectors the different linear transformations are used, $$ Q = XW_Q,\,K = XW_K,\,V = XW_V; W_Q \neq W_K, W_K \neq W_V, W_Q \neq W_V $$ The self-attention itself is a way of using more general attention mechanism. You can check this post for examples of other mechanisms that can be used.

Multi-head attention is a way of grouping together a bunch of attention mechanism ( Usually they are all the same type ), which consists in just running multiple mechanism in parallel and aggregating the resulting set in some way. In here for example, the aggregation is done by concatenation and weighted sum of the outputs, e.g. \begin{equation*} \begin{aligned} &\text{MultiHead}(Q, K, V)= [\text{head}_1, .., \text{head}_n]W^O & [..] \text{ meaning concatenation}\\ &\text{where head}_i = \text{Attention}(QW_i^Q,KW_i^K,VW_i^V) & \text{each head}_i\text{ is unique attention mechanism} \\ &\text{Attention}(\textbf{Q},\textbf{K},\textbf{V}) = \text{softmax}(\frac{\textbf{Q}\textbf{K}}{\sqrt{d_k}})\textbf{V} & \text{this can be other attention mechanism}\\ \end{aligned} \end{equation*}

  • 1
    $\begingroup$ are you sure that they are different? In the following document people.tamu.edu/~sji/classes/attn-slides.pdf (slide 5) it is clearly stated that *in self-attention, we let Q=K=V=X * $\endgroup$ Mar 7 at 11:42
  • $\begingroup$ Thanks for the comment, i checked it out. I think what happens is a misunderstanding in what Q, K, V are. In the slides you have provided they are seen as already transformed X that is passed to attention mechanism and transformed once again, kind of in the same way as I have established i-th head in the multi-head attention. $\endgroup$
    – vl_knd
    Mar 8 at 12:03
  • $\begingroup$ But since they are transformed again after being passed to the self attention, it is actually equivalent to what I have described as self attention. The only difference is that its applied to pre-transformed X. Imagine, that we are pre-transforming X to X*W. Now by applying the self attention I have described you are going to obtain the exact same result as in the slide 3 you have mentioned. I adopted this point of view, since this is the way its described in Attention is All you need paper. Check this for an example towardsdatascience.com/illustrated-self-attention-2d627e33b20a $\endgroup$
    – vl_knd
    Mar 8 at 12:21

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .