I have been looking into transformers lately and have been reading tons of tutorials. All of them address the intuition behind attention, which I understand. However, they treat learning the weight matrices (for query, key, and value) as it is the most trivial thing.

So, how are these weight matrices learned? Is the error just backpropagated, and the weights are updated accordingly?



You must log in to answer this question.

Browse other questions tagged .