New answers tagged attention
1
vote
Why are embeddings added, not concatenated?
First of all, I think it is very hard to properly reason about these things, but there are a few points that might justify using sum instead of concatenation.
For example, concatenation would have the ...
Top 50 recent answers are included
Related Tags
attention × 101transformer × 56
deep-learning × 31
neural-networks × 30
natural-language-processing × 30
recurrent-neural-networks × 15
machine-learning × 11
papers × 10
convolutional-neural-networks × 9
bert × 9
long-short-term-memory × 7
computer-vision × 6
comparison × 5
gpt × 5
pytorch × 4
python × 3
reference-request × 3
word-embedding × 3
convolution × 3
weights × 3
geometric-deep-learning × 3
machine-translation × 3
positional-encoding × 3
training × 2
terminology × 2