In articles that describe neural architectures with multiple attention layers of the same form, are the weight matrices usually the same across the layers? Consider as an example, "Attention is all you need". The authors stack several layers of multi-head self-attention in which each layer has the same number of heads. Each head $i$ involves a trainable weight matrix $W_{i}^{Q}$. There is no subscript, superscript, or any other indication that this matrix is different for each layer. My questions is this: are there separate $W_{i}^{Q}$ for layers $1,2,3,...$ or is this a single matrix shared throughout layers?
My intuition is that the authors of the paper wanted to cut down on notation, and that the matrices are different in different layers. But I want to be sure I understand this, since I see the same kind of thing in many other papers as well.
tensor2tensor
, as stated in the paper (if I remember correctly: please, check the footnotes of the paper, I think they say it there somewhere). Maybe it's this one: github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/…. You can check what they did there. You can then later provide an answer to your own question below ;) $\endgroup$