# In layman terms, what does "attention" do in a transformer?

I heard from many people about the paper titled Attention Is All You Need by Ashish Vaswani et al.

What actually does the "attention" do in simple terms? Is it a function, property, or some other thing?

## 2 Answers

Let's start by stressing out that in the literature unfortunately the term attention is still used widely without any precise consensus around the technical details, the only constant across papers is that attention should be used when a model is capable of learning, or focusing on local vs global patterns in the data we use for training. And with "should be used" I simply refer to the fact that everyone like to feel eligible to write "Hey, we used attention!" simply because of the hype generated by the introduction of transformers by Vaswani et al.

Said that, I think up to this point the best expression to describe attention is:

A specific type of architecture

What do I mean by this: Vaswani et al. introduced the expression attention in the paper you cite with a new whole machine learning architecture, namely the transformers. In the paper, attention is used to refer to a specific set of layers, similarly we call residual blocks or dense blocks specific type of layers combinations that were introduced for convolutional neural networks. For me the is no difference at all between attention and the two above mentioned examples. The confusion around the use of this expression in my opinion arose from the fact that Vaswani et al. put a lot of emphasis on the final purpose of the new proposed model, i.e. capturing local similarities within sentences in machine translation.

One last consideration why I think that architecture is the best label for attention is that it include also type of attentions that are completely different from the multi-head attention module introduced by Vaswani et al, like architectures that leverage attention maps. Mathematically, attention maps and the multi-head attention module share nothing but the name, still, because conceptually they seems to fulfill the same purpose, we call them both attention, with the consequence that to avoid confusion, one should always refer to a specific paper when talking about attention.

• This answer is good (I think, although I am not an expert in this topic), but there's one thing that may not be clear from your explanation. If "attention" is best described as a "layer" or "specific architecture" (submodule of a neural network), then, conceptually, what exactly does it do? (I think this was the question). You say "because conceptually they seems to fulfill the same purpose". What is the purpose then? That's one thing that is not emphasized enough in your answer.
– nbro
Oct 19 at 14:28
• I think that, before the transformer, attention was already gaining "attention" (sorry for the pun) within the machine translation community (see neural machine translation, for instance), from what I remember. The transformer paper certainly made the name "attention" and all related ideas even more famous, but I think it's important to note/emphasize that the idea of attention was already in the air before the transformer.
– nbro
Oct 19 at 14:29
• You should probably also clarify this sentence "For me the is no difference at all between attention and the two above mentioned examples" by saying that there's no difference in terms of role in a neural network (I think that's what you mean), i.e. that "attention" is just a layer or a module of a neural network, in the same way that "skip connections" are just specific types of layers or submodules.
– nbro
Oct 19 at 14:29

The answer above is very concise but I will try to give an ELI5 example. I also agree with @nbro that attention does not exclusively mean transformer architecture.

### Before attention

What is the height of the youngest female child of the father of your mother's first cousin? That query is convoluted, depends on your good memory of your family tree's relationships. It looks like this:

$$Q_{RNN} = f(Mother(first\_cousin(father(yongest\_female(height))))\ \ (1)$$

What you see in (1) is how Recurrent Neural Networks function, and it is obvious that their performance is inversely correlated with the length of the input.

### After attention

A more efficient way to query the height of that person is by giving it a name - Aligna. Now, whenever you need to get characteristics of that person you don't have to remember it's "chained" relationships backwards from the person that is closest to you. This new approach would look like this:

$$Q_{attentive_RNN} = f(Aligna(height))\ \ (2)$$

The latter is what essentially attention pooling does. Attention establishes a direct link with all input steps (e.g. words in an input sentence) so that it can independently pay attention to the input (or set of inputs) that will generate a successful output/prediction.