I have been looking into transformers lately and been reading tons of tutorials. All of them address the intuition behind attention, which I understand, but they treat training the weight matrices for query, key and value like it is the most trivial thing? So how are they trained? Is the error just backpropagated the weights are updated accordingly?

  • $\begingroup$ Hello. Welcome to AI SE! Have you already tried to look at any implementation of the transformer? I don't really have practical experience with the transformer (I only read the paper), but I think these matrices are usually trained in the same way that any other parameter of the transformer is trained, i.e. gradient descent and back-propagation (as you are correctly guessing). Maybe have a look at this implementation, where these matrices are just dense layers. $\endgroup$
    – nbro
    Sep 28 '20 at 21:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.