I am reading the book: Natural Language Processing with Transformers. It has the following paragraph

Although head_dim does not have to be smaller than the number of embedding dimensions of the tokens (embed_dim), in practice it is chosen to be a multiple of embed_dim so that the computation across each head is constant. For example, BERT has 12 attention heads, so the dimension of each head is 768/12=64.

While learning transformers, I tried to draw an analogy between CNN filters and multi-headed attention. For instance, increasing the number of filters helps learn different image features, while increasing number of heads help better understand the semantics of a sentence. However, it seems like that the input to the transformer (after being converted to embeddings) is being divided across heads. Perhaps my understanding of multi-head attention is incorrect.

Basically, I want to know why the author is dividing the inputs across the heads rather than feeding all the inputs to it.


1 Answer 1


You can give the entire modified inputs (q,k,v) to each of the heads. But to make it computationally faster, you make the modified inputs' sizes as num_heads * q,k,v and split them. It is the same thing.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .