Are multi-head attention matrices weighted adjacency matrices?

The job of the multi-head-attention mechanism in transformer models is to determine how likely a word is to appear after another word. In a sense this makes the resulting matrix a big graph with nodes and edges, where a node represents a word and an edge the likelihood to appear after that. So basically it is an adjacency matrix that is created.

  • $\begingroup$ Hello. Please, edit your post to provide more context (this post has been flagged as "needs more details") and explain why you're asking this question (what confuses you?). $\endgroup$
    – nbro
    Oct 13 '21 at 11:57
  • $\begingroup$ No, that isn't the job of the multi-head-attention mechanism. $\endgroup$
    – user253751
    Oct 18 '21 at 10:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.