1 vote

why we use learnable positional encoding instead of Sinusoidal positional encoding

transformers models use the sinusoidal position encoding to capture the position of each token, and this method has some limits : the sinusoidal position encoding is manually designed so it is not ...
LAILA EL OUEDEGHYRY's user avatar
1 vote

How the Q,K,V be calculated in multi-head attention

As far as I understand your question, you have problems with multiple heads. Let's take the input $\mathbf{Q}$ (with dimension: $\textit{seq_length}$ x $d_{model}$), which is the same for $\mathbf{Q}, ...
Dave's user avatar
  • 145

Only top scored, non community-wiki answers of a minimum length are eligible