1
$\begingroup$

In models using attention (eg Transformer architectures) we used scaled dot-product to measure similarity rather than (negative or inverse) Euclidean distance. Why is this the case?

Does Layer Normalization make these basically identical anyways? Or are there statistical, geometric, or training dynamics problems caused by treating embeddings as locations rather than directions? Or is this choice due to the extra computational overhead of using Euclidean distance?

$\endgroup$

2 Answers 2

3
$\begingroup$

Because it's a good fit, pretty much

Yes, you can use any "inverse distance measure", however, scalar product paired with softmax is a out-of-the-box good fit, as two vectors the less they agree (direction-wise), the "more negative" the scalar product gets, so $e^x$ with $x$ being the scalar product gets to 0, and the more the agree (in the direction), the "more positive" they get, so $e^x$ gets bigger and bigger

Bu sure, you can define other attention metrics, just pay attention to make them well behaved (so, no $\frac{1}{k^Tq}$)

Edit:
If you still want to use softmax to calculate attentions at the end, the logits has to be inversely proportional to the distance, because you want "high positive" values if they agree, and "low" values if they disagree

$\endgroup$
1
$\begingroup$

Dot-product attention and Euclidean distance-based attention are not necessarily similar, differing only in terms of computational cost. I wrote a recent paper to report that using the Euclidean distance can produce substantially different parameters and different behavior. This is due to the fact that, "while there exist order-preserving transformations between the Euclidean distance and the inner product, these transformations are not trivial; for each direction, the transformation involves adding one additional dimension [Bachrach et al., 2014]." Notably, "because Euclidean distance measures dissimilarity, it needs to be massaged into use as a similarity score." And I presented results which suggest that the negative-log of Euclidean distance is more promising than the negative Euclidean distance, and that the negative-log interestingly recovers "the inverse distance weighting function employed by Shepard [1968] for numerical interpolation of irregularly-spaced points."

C McCarter, “Inverse distance weighting attention.” Associative Memory & Hopfield Networks Workshop @ NeurIPS, 2023.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .