43
votes
Accepted
Why does the transformer do better than RNN and LSTM in long-range context dependencies?
I'll list some bullet points of the main innovations introduced by transformers , followed by bullet points of the main characteristics of the other architectures you mentioned, so we can then ...
- 4,283
19
votes
What is the intuition behind the dot product attention?
Let's start with a bit of notation and a couple of important clarifications.
$\mathbf{Q}$ refers to the query vectors matrix, $q_i$ being a single query vector associated with a single input word.
$\...
- 4,283
17
votes
Accepted
How is BERT different from the original transformer architecture?
What is a transformer?
The original transformer, proposed in the paper Attention is all you need (2017), is an encoder-decoder-based neural network that is mainly characterized by the use of the so-...
- 35k
16
votes
Accepted
How can Transformers handle arbitrary length input?
Actually, there is usually an upper bound for inputs of transformers, due to the inability of handling long-sequence. Usually, the value is set as 512 or 1024 at current stage.
However, if you are ...
- 321
11
votes
Accepted
What exactly are the "parameters" in GPT-3's 175 billion parameters and how are they chosen/generated?
Parameters is a synonym for weights, which is the term most people use for a neural networks parameters (and indeed in my experience it is a term that machine learners will use in general whereas ...
- 4,197
9
votes
Why does the transformer do better than RNN and LSTM in long-range context dependencies?
Let's start with RNN. A well known problem is vanishin/exploding gradients, which means that the model is biased by most recent inputs in the sequence, or in other words, older inputs have practically ...
- 276
8
votes
Accepted
What kind of word embedding is used in the original transformer?
I have found a good answer in this blog post The Transformer: Attention Is All You Need:
we learn a “word embedding” which is a smaller real-valued vector representation of the word that carries some ...
- 435
7
votes
Can the decoder in a transformer model be parallelized like the encoder?
Can the decoder in a transformer model be parallelized like the encoder?
The correct answer is: computation in a Transformer decoder can be parallelized during training, but not during actual ...
- 321
7
votes
Accepted
Can the decoder in a transformer model be parallelized like the encoder?
Can the decoder in a transformer model be parallelized like the
encoder?
Generally NO:
Your understanding is completely right. In the decoder, the output of each step is fed to the bottom decoder in ...
- 419
6
votes
Accepted
What is the purpose of Decoder mask (triangular mask) in Transformer?
The Transformer model presented in this tutorial is an auto-regressive Transformer. Which means that prediction of next token only depends on it's previous tokens.
So in order to predict next token, ...
- 230
6
votes
Accepted
What is different in each head of a multi-head attention mechanism?
The reason each head is different is because they each learn a different set of weight matrices $\{ W_i^Q, W_i^K, W_i^V \}$ where $i$ is the index of the head. To clarify, the input to each attention ...
- 1,260
6
votes
Accepted
In Computer Vision, what is the difference between a transformer and attention?
The original transformer is a feedforward neural network (FFNN)-based architecture that makes use of an attention mechanism. So, this is the difference: an attention mechanism (in particular, a self-...
- 35k
5
votes
What is the purpose of Decoder mask (triangular mask) in Transformer?
We give the target input into the transformer decoder while training the model. So it is easy for the model to "peek ahead" and learn what the next word would be. To ensure that this doesn't ...
- 151
5
votes
Why does GPT-2 Exclude the Transformer Encoder?
GPT-2 is a close copy of the basic transformer architecture.
GPT-2 does not require the encoder part of the original transformer architecture as it is decoder-only, and there are no encoder attention ...
- 625
4
votes
Accepted
Why does this multiplication of $Q$ and $K$ have a variance of $d_k$, in scaled dot product attention?
In statistics, if $X$ and $Y$ are independent and randomly distributed variables:
$\mathbb{E}[X + Y] = \mathbb{E}[X] + \mathbb{E}[Y] \\
Var(X + Y) = Var(X) + Var(Y) \\
\mathbb{E}[XY] = \mathbb{E}[X]\...
- 1,260
4
votes
Accepted
What is the cost function of a transformer?
I took a look at the Tensor2Tensor's source code implementation, and it seems like the loss function is the cross-entropy between the predicted probability matrix $\|\text{sentence length}\| \times \|\...
- 1,260
4
votes
In layman terms, what does "attention" do in a transformer?
Let's start by stressing out that in the literature unfortunately the term attention is still used widely without any precise consensus around the technical details, the only constant across papers is ...
- 4,283
4
votes
Accepted
What are the major layers in a Vision Transformer?
The Transformer family of architectures is a separate family of NN architectures, different from the CNNs and RNNs.
The main part of the Vision Transformer are the self-attention layers.
The ...
- 1,894
3
votes
Accepted
Any comparison between transformer and RNN+Attention on the same dataset?
If you go through the main introductory paper of the transformer ("Attention is all you need"), you can find the comparison of the model with other state-of-the-art machine translation ...
- 1,683
3
votes
What is the purpose of Decoder mask (triangular mask) in Transformer?
the mask is needed to prevent the decoder from "peeking ahead" at ground truth during training, when using its Attention mechanism.
Encoder:
Both runtime or training:
the encoder will ...
- 210
3
votes
Why would you implement the position-wise feed-forward network of the transformer with convolution layers?
I'm going to post another guess to this question - it won't be a complete answer, but hopefully it'll provide some direction towards finding a more legitimate answer.
The feed-forward networks as ...
- 81
3
votes
Accepted
What is the difference between the positional encoding techniques of the Transformer and GPT?
The purpose of introduction of positional encoding is to insert a notion of location of a given token in the sequence. Without it, due to the permutation equivariance (symmetry under the token ...
- 1,894
3
votes
Accepted
Transformers: how to get the output (keys and values) of the encoder?
I have read the OpenNMT source code (https://github.com/OpenNMT/OpenNMT-py/blob/cd29c1dbfb35f4a2701ff52a1bf4e5bdcf02802e/onmt/modules/multi_headed_attn.py).
It seems like an extra linear layer learns ...
- 1,260
3
votes
What kind of word embedding is used in the original transformer?
No, neither Word2Vec nor GloVe is used as Transformers are a newer class of algorithms. Word2Vec and GloVe are based on static word embeddings while Transformers are based on dynamic word embeddings.
...
- 1,503
3
votes
Accepted
How are certain machine learning models able to produce variable-length outputs given variable-length inputs?
In short, repetition with feedback.
You are correct that machine learning (ML) models such as neural networks work with fixed dimensions for input and output. There are a few different ways to work ...
- 24.7k
3
votes
Accepted
Is it realistic to train a transformer-based model (e.g. GPT) in a self-supervised way directly on the Mel spectrogram?
The reason most music-generation models use discrete representations is because the long-term structures of music are very challenging to model. Note that the MIDI data in MAESTRO (used in the two ...
- 194
3
votes
Accepted
Why do the authors of the T5 paper say that the "architectural changes are orthogonal to the experimental factors"?
"Orthogonal" is often used to mean "independent", as in "independent variable which does not correlate with the other variables". I believe this terminology originates ...
- 5,167
3
votes
What is the Intermediate (dense) layer in between attention-output and encoder-output dense layers within transformer block in PyTorch implementation?
Feedforward layer is an important part of the transformer architecture.
Transformer architecture, in addition to the self-attention layer, that aggregates ...
- 1,894
3
votes
Accepted
Do Vision Transformers handle arbitrary sequence lengths the same way as normal Transformers?
Yes, they can handle sequences with arbitrary length sequence, but with some remarks.
In the paper Training data-efficient image transformers & distillation through attention authors train models ...
- 1,894
3
votes
Accepted
Why do Transformers have a sequence limit at inference time?
Transformer models have limited sequence length at inference time
because of positional embeddings. But there are workarounds.
Self-attention in transformer does not distinguish the order of keys/...
- 66
Only top scored, non community-wiki answers of a minimum length are eligible