0
$\begingroup$

I’m looking to bridge two implementations of multihead attention.

Approach 1: Multiply and Split

Each of the queries, keys, and values is multiplied by a separate square weight matrix of size (embedding size, embedding size). The transformed embeddings are then split into $h$ number of subsets (heads), where each subset has a dimension of (embedding size / h).

Approach 2: Direct Projection

Each of the queries, keys, and values is multiplied by a weight matrix of size (embedding size, $d_k$), where $d_k$ is the dimension of each head, and typically $d_k = \text{embedding size} / h$. The projected embeddings are then reshaped into $h$ heads, each of dimension $d_k$.

Question

Are these equivalent mathematical implementations with differing efficiencies?

I’d like to grasp both the math and the PyTorch implementations here. There are similar answers here, but no one goes into explaining the differences in efficiency or the mathematical similarities between both.

$\endgroup$
1
  • $\begingroup$ Hello. Our site supports mathjax, so you can format the math symbols by wrapping them with $. Example: $d_k$ = $d_k$. $\endgroup$
    – nbro
    Commented May 28 at 15:09

0

You must log in to answer this question.