Timeline for Why is dot product attention faster than additive attention?
Current License: CC BY-SA 4.0
7 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jan 6, 2021 at 23:38 | history | edited | nbro | CC BY-SA 4.0 |
deleted 31 characters in body
|
| Sep 23, 2020 at 0:46 | comment | added | Jiang Xiang | Just think about the parameterizations that lead to Q and K. If they are obtained by some transformations such as Q=f(x; W_Q), K=f(x; W_K), there will be no need to apply additional transformations like Q=f(x; W_Q, A_Q), K=f(x; W_K, A_K). | |
| Apr 18, 2019 at 21:50 | comment | added | user3180 | That line still doesn't make sense to me. Looking at a reputable implementation of additive attention, they literally just sum the Q + K tensors. Are you implying the network takes in Q concatenated with K, passes through the MLP, to get Q + K? It does not make much sense to call that concatenation + MLP operation as "addition" | |
| Apr 18, 2019 at 20:21 | comment | added | John Doucette | @user3180 I think it's right in the quote in your question. " Additive attention computes the compatibility function using a feed-forward network with a single hidden layer. " A single hidden layer means it's a 3 layer network (input + hidden + output) | |
| Apr 18, 2019 at 16:41 | comment | added | user3180 | Lilian says the only difference between additive and dot product is Q + K vs Q K^T. I don't see any mention of multiple layers to compute Q + K | |
| Apr 18, 2019 at 16:41 | comment | added | user3180 | Do you have a source for the 3 layer structure? If you see the chart comparing attentions here: lilianweng.github.io/lil-log/2018/06/24/… | |
| Apr 18, 2019 at 4:00 | history | answered | John Doucette | CC BY-SA 4.0 |