I've been looking into self-attention lately, and in the articles that I've been seeing, they all talk about "weights" in attention. My understanding is that the weights in self-attention are not the same as the weights in a neural network.
From this article, http://peterbloem.nl/blog/transformers, in the additional tricks section, it mentions,
The query is the dot product of the query weight matrix and the word vector,
ie, q = W(q)x and the key is the dot product of the key weight matrix and the word vector,
k = W(k)x and similarly for the value it is
v = W(v)x. So my question is, where do the weight matrices come from?