As far as I know, attention was first introduced in Learning To Align And Translate.
There, the core mechanism which is able to disregard the sequence length, is a dynamically-built matrix, of shape output_size X input_size, in which every position $(o, i)$ holds the (log) probability that output $o$ should attend to input $i$.
That (log) probability is obtained by operating a learned function $a(h, s)$, where $h$ is a hidden state of the input, and $s$ is a cell state of the output.
Please let's disregard the fact that these inputs are RNN-based, and only look at the attention mechanism itself - a dynamic matrix of (log) probabilities is built, each slot is built by a function taking in two vectors, and outputting their "correspondence".
Jump forward to the iconic Attention Is All You Need.
Please disregard the fact that in this paper, $K$ was separated from $V$, unlike in the previous one.
I just want to look at the mechanism itself.
Let's look only at Multi-Head Attention, and in it, let's look only at the part actually doing the attention: $ QK^T $
Let's assume $Q$ and $K$ are vectors and not matrices, for simplicity. Their attention score is their dot product.
Let's compare the core attention mechanisms of "align and translate" against "all you need".
In "align and translate", the function learns how two vectors correspond to one another
In "all you need, the function learns to project embeddings into a continuous space, where they can be compared against other such projections by their dot-product.
One could easily implement multi-head-attention with the dynamic matrix method, by a function $b(k, q)$ yielding the (log) probability that the two correspond, and putting that into a dynamic-size matrix.
My question is what in the "all you need" core attention method makes it better than the "align and translate" core attention method?
Are there ablation studies for this point?
My intuition tells me it would be easier for a network to learn how to correspond vectors, rather than to learn an entire continuous space.
Again, please disregard the other contributions in "all you need", such as self-attention, separation of key from value, normalization, Transformer, ect.