2
$\begingroup$

I have been looking into transformers lately and have been reading tons of tutorials. All of them address the intuition behind attention, which I understand. However, they treat learning the weight matrices (for query, key, and value) as it is the most trivial thing.

So, how are these weight matrices learned? Is the error just backpropagated, and the weights are updated accordingly?

Improve this question
$\endgroup$
1
  • $\begingroup$ Short answer to an old question. Yes, the error is backpropagated and the weights are updated accordingly. The skip/residual connections in the transformer help to avoid vanishing gradients with larger amounts of decoder blocks in sequence. See also the 'basic training' section of this post. $\endgroup$
    – Robin van Hoorn
    Commented May 10, 2023 at 9:55

0

Reset to default

You must log in to answer this question.