# Why does this multiplication of $Q$ and $K$ have a variance of $d_k$, in scaled dot product attention?

In scaled dot product attention, we scale our outputs by dividing the dot product by the square root of the dimensionality of the matrix:

The reason why is stated that this constrains the distribution of the weights of the output to have a standard deviation of 1.

For example, consider that $$Q$$ and $$K$$ have a mean of 0 and variance of 1. Their matrix multiplication will have a mean of 0 and variance of $$d_k$$. Hence, square root of $$d_k$$ is used for scaling (and not any other number) because the matmul of $$Q$$ and $$K$$ should have a mean of 0 and variance of 1, and you get a gentler softmax.

Why does this multiplication have a variance of $$d_k$$?

If I understand this, I will then understand why dividing by $$\sqrt({d_k})$$ would normalize to 1.

Trying this experiment on 2x2 arrays I get an output of 1.6 variance:

## 1 Answer

It might help to take two small matrices that match the assumptions (mean of zero and variance of one) and just do the matrix multiplication. The dimensionality of K scales Q in the multiplication, scaling the variance simultaneously.

• Hey, I did do this but it does not come out as expected. I have edited the post – Jacob B May 21 at 1:09