I've been watching a few lectures on transformers, especially for language translation, though it seemingly becomes more confusing the more I watch.
In this lecture, there seems to be two conflicting views of self-attention. First, there's an Iron Man example (at around 44:25) where the lecturer claims that self-attention helps identify the important aspects of the input, but she details the math in which a "self-attention heat map", which is the dot product of the query and key matrices, is multiplied by a value matrix giving an output, which I assume is an attention head. It seems like this attention head just encodes how words are related to each other (the map) and the words themselves (the value).
I don't understand how this extracts the important information, or why you only want the important information anyways for translation. Is it extracting the important relationships between words, because that would make more sense, but in what way does that relate to her Iron Man example then? What's the relationship of Iron Man to himself? Also, would each attention head contain its own set of three matrices? Is the idea perhaps that words that more words depend on are important? But wouldn't that simply remove single words that still are important for translation?
TLDR: Essentially, how does the idea of relationships between words translate to selecting important words in a sentence (if this is even what self-attention does).