4
$\begingroup$

I am reading the book: Natural Language Processing with Transformers. It has the following paragraph

Although head_dim does not have to be smaller than the number of embedding dimensions of the tokens (embed_dim), in practice it is chosen to be a multiple of embed_dim so that the computation across each head is constant. For example, BERT has 12 attention heads, so the dimension of each head is 768/12=64.

While learning transformers, I tried to draw an analogy between CNN filters and multi-headed attention. For instance, increasing the number of filters helps learn different image features, while increasing number of heads help better understand the semantics of a sentence. However, it seems like that the input to the transformer (after being converted to embeddings) is being divided across heads. Perhaps my understanding of multi-head attention is incorrect.

Basically, I want to know why the author is dividing the inputs across the heads rather than feeding all the inputs to it.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

You can give the entire modified inputs (q,k,v) to each of the heads. But to make it computationally faster, you make the modified inputs' sizes as num_heads * q,k,v and split them. It is the same thing.

Improve this answer
$\endgroup$
2
  • $\begingroup$ What does the author mean by 768/12? Sorry, I am getting a little confused here. $\endgroup$
    – desert_ranger
    Commented Dec 21, 2022 at 2:30
  • $\begingroup$ Take a look at this image github.com/markriedl/transformer-walkthrough/blob/main/… (large image), see the encoder part first. You will understand the concept. $\endgroup$
    – p0p4k
    Commented Dec 21, 2022 at 8:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .