Difference between dot product attention and "matrix attention"

Question

As far as I know, attention was first introduced in Learning To Align And Translate.

There, the core mechanism which is able to disregard the sequence length, is a dynamically-built matrix, of shape output_size X input_size, in which every position $(o, i)$ holds the (log) probability that output $o$ should attend to input $i$.

That (log) probability is obtained by operating a learned function $a(h, s)$, where $h$ is a hidden state of the input, and $s$ is a cell state of the output.

Please let's disregard the fact that these inputs are RNN-based, and only look at the attention mechanism itself - a dynamic matrix of (log) probabilities is built, each slot is built by a function taking in two vectors, and outputting their "correspondence".

---

Jump forward to the iconic Attention Is All You Need.

Please disregard the fact that in this paper, $K$ was separated from $V$, unlike in the previous one.
I just want to look at the mechanism itself.

Let's look only at Multi-Head Attention, and in it, let's look only at the part actually doing the attention: $ QK^T $

Let's assume $Q$ and $K$ are vectors and not matrices, for simplicity. Their attention score is their dot product.

---

Let's compare the core attention mechanisms of "align and translate" against "all you need".

In "align and translate", the function learns how two vectors correspond to one another

In "all you need, the function learns to project embeddings into a continuous space, where they can be compared against other such projections by their dot-product.

One could easily implement multi-head-attention with the dynamic matrix method, by a function $b(k, q)$ yielding the (log) probability that the two correspond, and putting that into a dynamic-size matrix.

---

My question is what in the "all you need" core attention method makes it better than the "align and translate" core attention method?

Are there ablation studies for this point?

My intuition tells me it would be easier for a network to learn how to correspond vectors, rather than to learn an entire continuous space.

---

Again, please disregard the other contributions in "all you need", such as self-attention, separation of key from value, normalization, Transformer, ect.

Answer 1

The key difference between the attention mechanisms used in "Learning To Align And Translate" and "Attention Is All You Need" lies in the way that the similarity between the query and key vectors is calculated.

"Learning To Align And Translate"

The attention score is calculated by a learned function using a feed-forward neural network that takes in the query and key vectors and outputs a (log) probability of correspondence between them. This approach requires the model to learn a mapping from the input and output spaces to a joint space where the vectors can be compared against each other.

"Attention Is All You Need"

Here the attention is calculated as the similarity between the query and key vectors by taking their dot product and scaling it by the square root of their dimensionality. This approach does not require the model to learn a mapping to a joint space, but instead relies on the inherent structure of the vector space.

Pros/Cons

One advantage of the approach used in "Attention Is All You Need" is that it is computationally more efficient than the method used in "Learning To Align And Translate", especially for long sequences. Specifically, the scaled dot product attention is faster compared to "general/Bahdanau attention" in the sense that the latter is a learnt via a usually shallow feedforward neural network. In that sense, overhead space and time complexity is added while traversing the computational graph of the model as part of training.

That being said, there have been studies that have explored the use of different attention mechanisms in Transformers, including variants of the dot product and learned similarity functions. While the dot product attention used in "Attention Is All You Need" has shown to be effective in many cases, other mechanisms may be more appropriate for certain tasks or data types.

I copy below recent studies relevant to transformer variations and its attention mechanism.

• Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. CoRR, abs/1904.10509, 2019.
• Nikita Kitaev, L. Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. ArXiv, abs/2001.04451, 2020.
• Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, and T. Lillicrap. Compressive transformers for long-range sequence modelling. ArXiv, abs/1911.05507, 2020.
• Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. ArXiv, abs/2006.04768, 2020.
• A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention. In Proceedings of the International Conference on Machine Learning (ICML), 2020.
• Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng. Synthesizer: Rethinking self-attention in transformer models, 2020.