In the original transformer paper "Attention is all you need" in section 3.2.2 it is written:

Instead of performing a single attention function with dmodel-dimensional keys, values and queries, we found it beneficial to linearly project the queries, keys and values h times with different, learned linear projections to dk, dk and dv dimensions, respectively. On each of these projected versions of queries, keys and values we then perform the attention function in parallel, yielding dv-dimensional output values. These are concatenated and once again projected, resulting in the final values, as depicted in Figure 2.

I am wondering why you need to project the values h times if you concatenate and project them once again in the end. It seems to me that this just results in two matrices being learned that are multiplied with each other. This should have the same expressiveness as one matrix. So, in my understanding you could just leave away the h projection steps and simply do the final projection. Am I missing something?


1 Answer 1


This performs $h$ different attention lookups in parallel. The amount of computation is kept the same, though, so each one is smaller. They found that multiple smaller lookups were more useful than one big lookup.

The matrices are not just split, projected and concatenated. They are split, projected, the attention lookup is performed on each set of pieces and then the results of the attention function are concatenated.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .