New answers tagged attention
2
The Attention is All you Need has this footnote at the passage motivating the introduction of the $1/\sqrt{d_k}$ factor:
To illustrate why the dot products get large, assume that the components of $q$ and $k$ are independent random variables with mean 0 and variance 1. Then their dot product, $q \cdot k = \sum^{d_k}_{i=1}q_ik_i$ has mean 0 and variance $d_k$...
1
You are talking about model parallelism. But, that's not the reason RNNs/LSTMs are not in vogue.
Imagine your ability to read the first line of a page and going on reading and still making connections to the first line until the end of the page.
Can RNNs/LSTMs do that? No.
Can Attention (i.e. Transformers) do it? Yes.
The reason is simple Attention is ...
0
To answer this, you need some constraints on the problem. Here are some sequences of numbers. No machine learning technique could be expected to learn all of them:
the odd numbers
the primes
numbers expressed in digits, but listed in alphabetical order of their name in German
numbers listed in the lexical order of the reverse of their representation in base ...
3
GPT-2 is a close copy of the basic transformer architecture.
GPT-2 does not require the encoder part of the original transformer architecture as it is decoder-only, and there are no encoder attention blocks, so the decoder is equivalent to the encoder, except for the MASKING in the multi-head attention block, the decoder is only allowed to glean information ...
Top 50 recent answers are included
