Skip to main content
Artificial Intelligence

Return to Question

added 37 characters in body
Source Link
nbro
nbro
  • 41.4k
  • 12
  • 114
  • 205

How are weight matrices in attention trainedlearned?

I have been looking into transformers lately and have been reading tons of tutorials. All of them address the intuition behind attention, which I understand. However, but they treat traininglearning the weight matrices for(for query, key, and value like) as it is the most trivial thing?.

So, how are they trainedthese weight matrices learned? Is the error just backpropagated, and the weights are updated accordingly?

How are weight matrices in attention trained?

I have been looking into transformers lately and been reading tons of tutorials. All of them address the intuition behind attention, which I understand, but they treat training the weight matrices for query, key and value like it is the most trivial thing? So how are they trained? Is the error just backpropagated the weights are updated accordingly?

How are weight matrices in attention learned?

I have been looking into transformers lately and have been reading tons of tutorials. All of them address the intuition behind attention, which I understand. However, they treat learning the weight matrices (for query, key, and value) as it is the most trivial thing.

So, how are these weight matrices learned? Is the error just backpropagated, and the weights are updated accordingly?

Source Link
Ege Demir
Ege Demir
  • 21
  • 1

How are weight matrices in attention trained?

I have been looking into transformers lately and been reading tons of tutorials. All of them address the intuition behind attention, which I understand, but they treat training the weight matrices for query, key and value like it is the most trivial thing? So how are they trained? Is the error just backpropagated the weights are updated accordingly?