What is the intuition behind self-attention?

Question

I've been watching a few lectures on transformers, especially for language translation, though it seemingly becomes more confusing the more I watch.

In this lecture, there seems to be two conflicting views of self-attention. First, there's an Iron Man example (at around 44:25) where the lecturer claims that self-attention helps identify the important aspects of the input, but she details the math in which a "self-attention heat map", which is the dot product of the query and key matrices, is multiplied by a value matrix giving an output, which I assume is an attention head. It seems like this attention head just encodes how words are related to each other (the map) and the words themselves (the value).

I don't understand how this extracts the important information, or why you only want the important information anyways for translation. Is it extracting the important relationships between words, because that would make more sense, but in what way does that relate to her Iron Man example then? What's the relationship of Iron Man to himself? Also, would each attention head contain its own set of three matrices? Is the idea perhaps that words that more words depend on are important? But wouldn't that simply remove single words that still are important for translation?

TLDR: Essentially, how does the idea of relationships between words translate to selecting important words in a sentence (if this is even what self-attention does).

Answer 1

First of all, I totally agree that the Iron Man example is a little off the topic and not really clear to explain the concept of self-attention.

But she had a point there. Just like what you said, the self-attention mechanism extracts the relationships between words and then after being scaled down and multiply with the value tensor, it makes more sense for what are the important keys that need to be attended by the model.

Now let's get back to the Iron Man example. I think the idea is that our brains can immediately know which parts are the important part of the image to focus on because we all have that attention filter that is always turning on.

It's different from the CNN or the RNN model, which needs to know the feature first (feature extraction). The self-attention calculates the relationship between each token and the importance on the whole image first and then can use that information to focus on feature extraction on those really important parts.