What is the weight matrix in self-attention?

I've been looking into self-attention lately, and in the articles that I've been seeing, they all talk about "weights" in attention. My understanding is that the weights in self-attention are not the same as the weights in a neural network.

From this article, http://peterbloem.nl/blog/transformers, in the additional tricks section, it mentions,

The query is the dot product of the query weight matrix and the word vector, ie, q = W(q)x and the key is the dot product of the key weight matrix and the word vector, k = W(k)x and similarly for the value it is v = W(v)x. So my question is, where do the weight matrices come from?

• Hi. I just wanted to let you know that you can use latex on this site.
– nbro
Aug 30, 2020 at 0:56

1 Answer

The answer is actually really simple: they are all randomly initialised. So they are to all intents and purposes "normal" weights of a neural network.

This is also the reason why in the original paper the authors tested several setting with single and multiple attention heads. If these matrices were somehow "special" or predetermined they would all serve the same purpose. Instead, because of their random initialisation, each attention head learn to contribute to solve a different task, like they show in Figure 3 and 4.

• So how are those weight matrices updated? Is there a separate neural network for that? Sep 1, 2020 at 2:16
• Why do you need another neural network? You just back propagate everything till these very first matrices. You can see from the tensorflow documentation that the matrices are just dense layers (or conv1D layers) github.com/tensorflow/tensor2tensor/blob/… Sep 1, 2020 at 10:55