Skip to main content

New answers tagged

1 vote

Why are the Q and K matrices two separate matrices in attention?

Your calculations are correct, though training a whole $W$ is not always equivalent. We can agree that $W_q, W_k \in R^{n\times m}$, thus $W_{qk} = W_qW_k^T \in R^{n\times n}$ Now, check the parameter ...
Alberto's user avatar
  • 2,283

Top 50 recent answers are included