Timeline for Why does this multiplication of $Q$ and $K$ have a variance of $d_k$, in scaled dot product attention?
Current License: CC BY-SA 4.0
7 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Apr 29, 2023 at 21:34 | comment | added | Peyman | I appreciate it if you answer this related question ai.stackexchange.com/questions/40244/… | |
| Dec 8, 2020 at 22:23 | comment | added | Jacob B | Hmmm I see I will try with larger shapes, in any case your answer provided great insight to my initial question so thanks for taking the time! | |
| Dec 8, 2020 at 20:53 | comment | added | user3667125 | Ah, you are right! dot is the same as matmul. I'm not sure how to explain the code part in that case. Maybe dimension=2 is too small? The sample variance will itself have a variance that reduces with larger dimension size. | |
| Dec 8, 2020 at 20:51 | history | edited | user3667125 | CC BY-SA 4.0 |
added 98 characters in body
|
| Dec 8, 2020 at 20:40 | comment | added | Jacob B | Awesome, thank you this is a great answer. I am still confused on the code part though looking at the docs for numpy the np.dot function should be equivalent to np.matmul in 2 dimensions. | |
| Dec 8, 2020 at 20:40 | vote | accept | Jacob B | ||
| Dec 8, 2020 at 2:07 | history | answered | user3667125 | CC BY-SA 4.0 |