What's the difference between content-based attention and dot-product attention?

Question

I'm following this blog post which enumerates the various types of attention.

It mentions content-based attention where the alignment scoring function for the $j$th encoder hidden state with respect to the $i$th context vector is the cosine distance:

$$ e_{ij} = \frac{\mathbf{h}^{enc}_{j}\cdot\mathbf{h}^{dec}_{i}}{||\mathbf{h}^{enc}_{j}||\cdot||\mathbf{h}^{dec}_{i}||} $$

It also mentions dot-product attention:

$$ e_{ij} = \mathbf{h}^{enc}_{j}\cdot\mathbf{h}^{dec}_{i} $$

To me, it seems like these are only different by a factor. If we fix $i$ such that we are focusing on only one time step in the decoder, then that factor is only dependent on $j$. Specifically, it's $1/\mathbf{h}^{enc}_{j}$.

So we could state: "the only adjustment content-based attention makes to dot-product attention, is that it scales each alignment score inversely with the norm of the corresponding encoder hidden state before softmax is applied."

What's the motivation behind making such a minor adjustment? What are the consequences?

---

Follow up question:

What's more, is that in Attention is All you Need they introduce the scaled dot product where they divide by a constant factor (square root of size of encoder hidden vector) to avoid vanishing gradients in the softmax. Any reason they don't just use cosine distance?

Answer 1

The Attention is All you Need has this footnote at the passage motivating the introduction of the $1/\sqrt{d_k}$ factor:

4. To illustrate why the dot products get large, assume that the components of $q$ and $k$ are independent random variables with mean 0 and variance 1. Then their dot product, $q \cdot k = \sum^{d_k}_{i=1}q_ik_i$ has mean 0 and variance $d_k$.

I suspect that it hints on the cosine-vs-dot difference intuition. The cosine similarity ignores magnitudes of the input vectors - you can scale $h^{enc}$ and $h^{dec}$ by arbitrary factors and still get the same value of the cosine distance.

The footnote talks about vectors with normally distributed components, clearly implying that their magnitudes are important. This suggests that the dot product attention is preferable, since it takes into account magnitudes of input vectors. And the magnitude might contain some useful information about the "absolute relevance" of the $Q$ and $K$ embeddings.

Edit after more digging: Note that transformer architecture has the Add & Norm blocks after each attention and FF block. At first I thought that it settles your question: since every input vector is normalized then cosine distance should be equal to the dot product. That's incorrect though - the "Norm" here means Layer Normalization - analogously to batch normalization it has trainable mean and scale parameters, so my point above about the vector norms still holds.

Finally, since apparently we don't really know why the BatchNorm works same thing holds for the LayerNorm. Interestingly, it seems like (1) BatchNorm (2) LayerNorm and (3) your question about normalization in the attention mechanism - all of it look like different ways at looking at the same, yet undiscovered and clearly stated thing.