# What is the weight matrix in self-attention?

I've been looking into self-attention lately, and in the articles that I've been seeing, they all talk about "weights" in attention. My understanding is that the weights in self-attention are not the same as the weights in a neural network.

The query is the dot product of the query weight matrix and the word vector, ie, q = W(q)x and the key is the dot product of the key weight matrix and the word vector, k = W(k)x and similarly for the value it is v = W(v)x. So my question is, where do the weight matrices come from?

• Hi. I just wanted to let you know that you can use latex on this site.
– nbro
Aug 30, 2020 at 0:56

The answer is actually really simple: they are all randomly initialised. So they are to all intents and purposes "normal" weights of a neural network.

This is also the reason why in the original paper the authors tested several setting with single and multiple attention heads. If these matrices were somehow "special" or predetermined they would all serve the same purpose. Instead, because of their random initialisation, each attention head learn to contribute to solve a different task, like they show in Figure 3 and 4.

• So how are those weight matrices updated? Is there a separate neural network for that?
– Mark
Sep 1, 2020 at 2:16
• Why do you need another neural network? You just back propagate everything till these very first matrices. You can see from the tensorflow documentation that the matrices are just dense layers (or conv1D layers) github.com/tensorflow/tensor2tensor/blob/… Sep 1, 2020 at 10:55

In my mind there are two weight matrices, the one you get prior to applying softmax:

$$\alpha_{i,j} = \frac{\langle q_i, k_j \rangle}{\sqrt{d}}$$

the other you get after applying the softmax:

$$\text{Attn}_{i,j}(X) = \frac{\exp \left( \frac{\langle q_i, k_j \rangle}{\sqrt{d}}\right)}{\sum_k \exp\left( \frac{\langle q_i, k_k \rangle}{\sqrt{d}} \right)}.$$

Either way, you can view them as directed "attention graphs" and they can be interpreted in terms of graph neural networks using only complete graphs and graph attention:

$$A_{i,j} = \frac{\exp(w(i,j))}{\sum_{j \in \mathcal{N}(i)} \exp(w(i,k))},$$

where $$w(i,j)$$ is a weight function, which can be the scaled dot-product of attention for example. The sum is over the neighborhood of a node (think token) $$\mathcal{N}(i)$$. Hope that helps a little with intuition and that it's close to what you meant.