I've been looking into self-attention lately, and in the articles that I've been seeing, they all talk about "weights" in attention. My understanding is that the weights in self-attention are not the same as the weights in a neural network.

From this article, http://peterbloem.nl/blog/transformers, in the additional tricks section, it mentions,

The query is the dot product of the query weight matrix and the word vector, ie, q = W(q)x and the key is the dot product of the key weight matrix and the word vector, k = W(k)x and similarly for the value it is v = W(v)x. So my question is, where do the weight matrices come from?

  • 1
    $\begingroup$ Hi. I just wanted to let you know that you can use latex on this site. $\endgroup$
    – nbro
    Aug 30, 2020 at 0:56

1 Answer 1


The answer is actually really simple: they are all randomly initialised. So they are to all intents and purposes "normal" weights of a neural network.

This is also the reason why in the original paper the authors tested several setting with single and multiple attention heads. If these matrices were somehow "special" or predetermined they would all serve the same purpose. Instead, because of their random initialisation, each attention head learn to contribute to solve a different task, like they show in Figure 3 and 4.

  • $\begingroup$ So how are those weight matrices updated? Is there a separate neural network for that? $\endgroup$
    – Mark Shaio
    Sep 1, 2020 at 2:16
  • 1
    $\begingroup$ Why do you need another neural network? You just back propagate everything till these very first matrices. You can see from the tensorflow documentation that the matrices are just dense layers (or conv1D layers) github.com/tensorflow/tensor2tensor/blob/… $\endgroup$ Sep 1, 2020 at 10:55

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .