0
$\begingroup$

Why do we call the attention layer in transformers multi-headed attention when in practice all the attention matrices from different heads (W,K,V) for a single layer are concatenated to perform the calculation in one go and then the result is multiplied by another matrix Wo to get the dimensions required?

My understanding of self-attention calculation is based on the following blog: https://jalammar.github.io/illustrated-transformer/

$\endgroup$
0

1 Answer 1

2
$\begingroup$

The original paper "Attention is all you need" mentions the following.

" Instead of performing a single attention function with $ d_{model} $-dimensional keys, values and queries, we found it beneficial to linearly project the queries, keys and values $h$ times with different, learned linear projections to $d_k$, $d_k$ and $d_v$ dimensions, respectively. "

What is single-headed attention?

The input matrix $ X $ of dimension $ T \times d_{model} $ is multipled with $ W^Q, W^K $ and $W^V $ matrices of dimension $ d_{model} \times d_{model} $ to give output of dimension $ T \times d_{model} $ Now, while calculating attention $ A=softmax(\frac{QK^{T}}{\sqrt{d_k}}) $yields one attention matrix of dimension $ T \times T $.

What is multi-headed attention?

The input matrix $ X $ of dimension $ T \times d_{model} $ is multipled with $ W^Q, W^K $ and $W^V $ matrices of dimension $ d_{model} \times d_{k} $ to give output of dimension $ T \times d_{k} $. Note that $d_{model} = h \times d_{k} $ where $ h $ is the number of heads. Now, while calculating attention $ A=softmax(\frac{QK^{T}}{\sqrt{d_k}}) $yields $h$ attention matrices of dimension $ T \times T $.

It is called multi-headed attention because each head produces one different attention matrix.

(I feel your confusion may have come from the knowledge that matrix multiplication happens in blocks. So what single-headed attention does in $ d_{model} \times d_{model} $ matrix, multi-headed attention does in $ h $ numbers of $ d_{model} \times d_{k} $ matrices. Why not make one single $ d_{model} \times d_{model} $ matrix from these $ h $ numbers of $ d_{model} \times d_{k} $ matrices? and proceed as in single-headed attention. The only problem with that argument is we end up with one attention matrix instead $h$ attention matrices. )

These $ h $ attention matrices help in modeling different relationships between words at different positions. The original paper mentions

"Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this."

$\endgroup$
3
  • $\begingroup$ I meant that single headed attention is in concept only and in actual implementation the Q, K, V blocks for a single layer is just a single matrix. For example if single Q matrix is 10x8 and there are 8 heads then the actual Q matrix is just 10x64. Same for K and V and the forward operations are applied. Then in the end we have seq_lenx64 which is multiplied by 64x8 to get the final dimensions correct. So why isn't it just considered a single weight instead of multi-headed? $\endgroup$
    – Tarique
    Nov 15, 2023 at 15:40
  • $\begingroup$ Work out the dimensions of your attention matrix. Imagine you have 10x8 Q,K, V matrices. Then dimension of your attention matrix is arrived as ($ KQ^T $ ) (10 x 8) x (8x10) = 10 x 10. That is the size of your attention matrix for a single head. Do it for 8 heads and you get 8 numbers of 10 x 10 attention matrices. Now , because these 10 x8 matrices can be treated as blocks of a single matrix, you have one 10 x 64 matrix for K. Same applies for Q and V. Now work out the dimensions of your attention matrix (10 x64) x (10 x 64) = 10 x 10. You get only one attention matrix of dimension 10 x10. $\endgroup$ Nov 16, 2023 at 2:03
  • 1
    $\begingroup$ Each attention matrix is used to generate a part of the output in multi-headed attention. All such outputs are concatenated to produce the final output. This is not the same output as one full sized output generated using one attention matrix in single headed case. So both in actual implementation and concept, multi-headed attention is different from single-headed attention. $\endgroup$ Nov 16, 2023 at 4:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .