Attention idea is one of the most influential ideas in deep learning. The main idea behind attention technique is that it allows the decoder to "look back” at the complete input and extracts significant information that is useful in decoding.

I am really having trouble understanding the intuition behind the attention mechanism. I mean how the mechanism works and how to configure.

In simple words (and maybe with an example), what is the intuition behind the attention mechanism?

What are some applications, advantages & disadvantages of attention mechanism?


1 Answer 1


Simply put, the attention mechanism is loosely inspired on well, attention. Consider we are attempting machine translation on the following sentence: "The dog is a Labrador." If you were to ask someone to pick out the key words of the sentence, i.e. which ones encode the most meaning, they would likely say "dog" and "Labrador." Articles like "the" and "a" are not as relevant in translation as the previous words (though they aren't completely insignificant). Therefore, we focus our attention on the important words.

Attention seeks to mimic this by adding attention weights to a model as trainable parameters to augment important parts of our input. Consider an encoder-decoder architecture such as the one Google Translate uses. Our encoder recurrent neural network (RNN) encodes our input sentence as a context vector in some vector space, which is then passed along to the decoder RNN which translates it into the target language. The attention mechanism scores each word in the input (via dot product with attention weights), then passes these scores through the softmax function to create a distribution. This distribution is then multiplied with the context vector to produce an attention vector, which is then passed to the decoder. In the example in the first paragraph, our attention weights for "dog" and "Labrador" would hopefully become larger in comparison to those for the other words during training. Note that all parts of the input are still considered since a distribution must sum to 1, just some elements have more effect on the output than others.

Below is a diagram from Towards Data Science that illustrates this concept very nicely in terms of an encoder-decoder architecture.

enter image description here

The advantages of attention is its ability to identify the information in an input most pertinent to accomplishing a task, increasing performance especially in natural language processing - Google Translate is a bidirectional encoder-decoder RNN with attention mechanisms. The disadvantage is the increased computation. In humans, attention serves to reduce our workload by allowing us to ignore unimportant features, however in a neural network, attention entails overhead as we are now generating attention distributions and training our attention weights (we are not actually ignoring the unimportant features, just diminishing their importance).

  • $\begingroup$ Clarification: The input embedding is NOT used in the attention mechanism. The Attention Scores (usually called Alignment scores) are computed between the attention weights and the encoder hidden state (not the input embedding!). Similarly, the attention output is a linear combination between the attention distribution and encoder hidden states. $\endgroup$ Oct 20, 2020 at 7:24

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .