In scaled dot product attention, we scale our outputs by dividing the dot product by the square root of the dimensionality of the matrix:

enter image description here

The reason why is stated that this constrains the distribution of the weights of the output to have a standard deviation of 1.

Quoted from https://www.tensorflow.org/tutorials/text/transformer:

For example, consider that $Q$ and $K$ have a mean of 0 and variance of 1. Their matrix multiplication will have a mean of 0 and variance of $d_k$. Hence, square root of $d_k$ is used for scaling (and not any other number) because the matmul of $Q$ and $K$ should have a mean of 0 and variance of 1, and you get a gentler softmax.

Why does this multiplication have a variance of $d_k$?

If I understand this, I will then understand why dividing by $\sqrt({d_k})$ would normalize to 1.

Trying this experiment on 2x2 arrays I get an output of 1.6 variance:

enter image description here


It might help to take two small matrices that match the assumptions (mean of zero and variance of one) and just do the matrix multiplication. The dimensionality of K scales Q in the multiplication, scaling the variance simultaneously.

| improve this answer | |
  • $\begingroup$ Hey, I did do this but it does not come out as expected. I have edited the post $\endgroup$ – Jacob B May 21 at 1:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.