I'm following this blog post which enumerates the various types of attention.

It mentions content-based attention where the alignment scoring function for the $j$th encoder hidden state with respect to the $i$th context vector is the cosine distance:

$$ e_{ij} = \frac{\mathbf{h}^{enc}_{j}\cdot\mathbf{h}^{dec}_{i}}{||\mathbf{h}^{enc}_{j}||\cdot||\mathbf{h}^{dec}_{i}||} $$

It also mentions dot-product attention:

$$ e_{ij} = \mathbf{h}^{enc}_{j}\cdot\mathbf{h}^{dec}_{i} $$

To me, it seems like these are only different by a factor. If we fix $i$ such that we are focusing on only one time step in the decoder, then that factor is only dependent on $j$. Specifically, it's $1/\mathbf{h}^{enc}_{j}$.

So we could state: "the only adjustment content-based attention makes to dot-product attention, is that it scales each alignment score inversely with the norm of the corresponding encoder hidden state before softmax is applied."

What's the motivation behind making such a minor adjustment? What are the consequences?

Follow up question:

What's more, is that in Attention is All you Need they introduce the scaled dot product where they divide by a constant factor (square root of size of encoder hidden vector) to avoid vanishing gradients in the softmax. Any reason they don't just use cosine distance?


1 Answer 1


The Attention is All you Need has this footnote at the passage motivating the introduction of the $1/\sqrt{d_k}$ factor:

  1. To illustrate why the dot products get large, assume that the components of $q$ and $k$ are independent random variables with mean 0 and variance 1. Then their dot product, $q \cdot k = \sum^{d_k}_{i=1}q_ik_i$ has mean 0 and variance $d_k$.

I suspect that it hints on the cosine-vs-dot difference intuition. The cosine similarity ignores magnitudes of the input vectors - you can scale $h^{enc}$ and $h^{dec}$ by arbitrary factors and still get the same value of the cosine distance.

The footnote talks about vectors with normally distributed components, clearly implying that their magnitudes are important. This suggests that the dot product attention is preferable, since it takes into account magnitudes of input vectors. And the magnitude might contain some useful information about the "absolute relevance" of the $Q$ and $K$ embeddings.

Edit after more digging: Note that transformer architecture has the Add & Norm blocks after each attention and FF block. At first I thought that it settles your question: since every input vector is normalized then cosine distance should be equal to the dot product. That's incorrect though - the "Norm" here means Layer Normalization - analogously to batch normalization it has trainable mean and scale parameters, so my point above about the vector norms still holds.

Finally, since apparently we don't really know why the BatchNorm works same thing holds for the LayerNorm. Interestingly, it seems like (1) BatchNorm (2) LayerNorm and (3) your question about normalization in the attention mechanism - all of it look like different ways at looking at the same, yet undiscovered and clearly stated thing.

  • $\begingroup$ I think it's a helpful point. I'll leave this open till the bounty ends in case any one else has input. Thanks. $\endgroup$ Commented Apr 18, 2021 at 21:42
  • $\begingroup$ @AlexanderSoare Thank you (also for great question). I've spent some more time digging deeper into it - check my edit. $\endgroup$
    – Kostya
    Commented Apr 18, 2021 at 23:25
  • $\begingroup$ Thanks for sharing more of your thoughts. I never thought to related it to the LayerNorm as there's a softmax and dot product with $V$ in between so things rapidly get more complicated when trying to look at it from a bottom up perspective. I think my main takeaways from your answer are a) cosine distance doesn't take scale into account, b) they divide by $sqrt(d_k)$ but it could have been something else and might have worked and we don't really know why $\endgroup$ Commented Apr 19, 2021 at 9:04
  • $\begingroup$ By the way, re layer norm vs batch norm I also have this question if you're interested in following it $\endgroup$ Commented Apr 19, 2021 at 9:04
  • $\begingroup$ github.com/neqkir/attention-mechanism $\endgroup$
    – kiriloff
    Commented Jan 14, 2022 at 11:57

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .