0
$\begingroup$

I want to understand the transformer architecture, so I start with self attention and I understand their mechanism, but when I pass to the multi-head attention I find some difficulties like how calculate Q , K and V for each head. I find many way to calculate Q , K and V but I don't know which way is correct.
method 1: this image illustrate how calcule these matrices method 2: I find this method in YouTube. enter image description here method 3: in this method it just for headi we have: Qi=xWqi Ki=xWki Vi=xWvi so I don't know which method is correct there is the links of my references The Illustrated Transformer YouTube video

$\endgroup$
5
  • $\begingroup$ Q,K,V are the input matrices. In self attention they would just be 3 copies of the input sequence(s) $\endgroup$
    – David
    Apr 2 at 10:05
  • $\begingroup$ so you man that for each headi we have: For the query matrix: Qi​=X×Wqi​ For the key matrix: Ki​=X×Wki For the value matrix: Vi​=X×Wvi which X in the input of encoder or the output of the previous encoder if we have maby layer of decoder $\endgroup$ Apr 2 at 10:47
  • $\begingroup$ Yes. Important to note that the project matrices W_{q,v,k} are different for each head. $\endgroup$
    – David
    Apr 2 at 10:57
  • 1
    $\begingroup$ Yes, I get it ,for each head we initialize randomly W_{q,v,k} $\endgroup$ Apr 2 at 11:54
  • $\begingroup$ Ill add this, which has an intuitive explanation on its calculation as well:ai.stackexchange.com/questions/40179/… $\endgroup$ Apr 7 at 10:26

2 Answers 2

1
$\begingroup$

As far as I understand your question, you have problems with multiple heads. Let's take the input $\mathbf{Q}$ (with dimension: $\textit{seq_length}$ x $d_{model}$), which is the same for $\mathbf{Q}, \mathbf{K}$ and $\mathbf{V}$ (thus $\mathbf{Q} = \mathbf{K} = \mathbf{V}$). then you need to multiply them with 3 different matrices (with dimension: $d_{model}$ x $d_{model}$), thus: $\mathbf{Q}$ x $\mathbf{W}_{q}$, $\mathbf{K}$ x $\mathbf{W}_{k}$, $\mathbf{V}$ x $\mathbf{W}_{v}$. Why this multiplication? Because those are the matrices with the learnable parameters. Now... in case of multi head attention, you need to split them by the number heads, that you want. Since the model dimension ($d_{model}$) must be divisible by the number of heads ($h$). At this point you do not need to create or use any complicate function, you can just reshape the three matrices in this way:

# (batch, seq_len, d_model) --> (batch, seq_len, h, d_k) --> (batch, h, seq_len, d_k)

or in PyTorch code if you prefer:

query = query.view(query.shape[0], query.shape[1], self.h, self.d_k).transpose(1, 2) (where d_k = dim_model // h)

In case you don't "see" the division in chunks, open a terminal, import torch and try the line of code above (removing the batch in all the calculation) allright?

$\endgroup$
4
  • $\begingroup$ so you say that we directly multiple the input x by Wq , Wk and Wv and then split them into a number of head that we want, we don't need for each head i to multiply x by Wiq because in the paper "attention is all you need" they say that each head i has as input QWiq and KWik and VWiv that's make me in confusion and don't understand which method is correct $\endgroup$ Apr 7 at 12:54
  • $\begingroup$ Forget the concept of "head" for a moment. Imagine only one block: an attention block. In this block you have one input. This input is copied three times (Q, K, and V). Each of them is then multiplied by a weight matrix. Stop. Once you have it, you can think about how split those three matrices in $h$ heads. $\endgroup$
    – Dave
    Apr 7 at 16:08
  • $\begingroup$ About the paper: once you multiply Q by its weight matrix, you get a new matrix $Q'$. which is exactly $Q' = Q W_{q}$. The same for K and V. Just Multiply Q, K and V by their respectively matrices and go on with the code $\endgroup$
    – Dave
    Apr 7 at 16:11
  • $\begingroup$ Look at this video. It has been published 3 hours ago: youtu.be/eMlx5fFNoYc?si=SlEtj84pUHh4gxBh $\endgroup$
    – Dave
    Apr 7 at 18:30
0
$\begingroup$

Given the input $x$ into the head, Q, K, and V are simply $x$, but then adapted by distinct fully connected neural network layer.

To clarify, each head has 3 linear layers, one for constructing Q, one for constructing K, and one for constructing V. The input for each of these linear layers is just a copy of $x$. Each head (of multi head attention) has their own set of parameters.

Notably, these learnable linear layers are the only learnable parameters of the head.

$\endgroup$
1
  • $\begingroup$ so you say that for each head i to multiply x by Wiq , etc and in the paper "attention is all you need" when they say that each head i has as input QWiq and KWik and VWiv they mean by Q,K,V the copy of the input? $\endgroup$ Apr 7 at 12:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .