In layman terms, what does "attention" do in a transformer?

Question

I heard from many people about the paper titled Attention Is All You Need by Ashish Vaswani et al.

What actually does the "attention" do in simple terms? Is it a function, property, or some other thing?

Answer 1

Let's start by stressing out that in the literature unfortunately the term attention is still used widely without any precise consensus around the technical details, the only constant across papers is that attention should be used when a model is capable of learning, or focusing on local vs global patterns in the data we use for training. And with "should be used" I simply refer to the fact that everyone like to feel eligible to write "Hey, we used attention!" simply because of the hype generated by the introduction of transformers by Vaswani et al.

Said that, I think up to this point the best expression to describe attention is:

A specific type of architecture

What do I mean by this: Vaswani et al. introduced the expression attention in the paper you cite with a new whole machine learning architecture, namely the transformers. In the paper, attention is used to refer to a specific set of layers, similarly we call residual blocks or dense blocks specific type of layers combinations that were introduced for convolutional neural networks. For me the is no difference at all between attention and the two above mentioned examples. The confusion around the use of this expression in my opinion arose from the fact that Vaswani et al. put a lot of emphasis on the final purpose of the new proposed model, i.e. capturing local similarities within sentences in machine translation.

One last consideration why I think that architecture is the best label for attention is that it include also type of attentions that are completely different from the multi-head attention module introduced by Vaswani et al, like architectures that leverage attention maps. Mathematically, attention maps and the multi-head attention module share nothing but the name, still, because conceptually they seems to fulfill the same purpose, we call them both attention, with the consequence that to avoid confusion, one should always refer to a specific paper when talking about attention.

Answer 2

The answer above is very concise but I will try to give an ELI5 example. I also agree with @nbro that attention does not exclusively mean transformer architecture.

---

Before attention

What is the height of the youngest female child of the father of your mother's first cousin? That query is convoluted, depends on your good memory of your family tree's relationships. It looks like this:

$Q_{RNN} = f(Mother(first\_cousin(father(yongest\_female(height))))\ \ (1)$

What you see in (1) is how Recurrent Neural Networks function, and it is obvious that their performance is inversely correlated with the length of the input.

After attention

A more efficient way to query the height of that person is by giving it a name - Aligna. Now, whenever you need to get characteristics of that person you don't have to remember it's "chained" relationships backwards from the person that is closest to you. This new approach would look like this:

$Q_{attentive_RNN} = f(Aligna(height))\ \ (2)$

The latter is what essentially attention pooling does. Attention establishes a direct link with all input steps (e.g. words in an input sentence) so that it can independently pay attention to the input (or set of inputs) that will generate a successful output/prediction.