38
votes
Accepted
Why does the transformer do better than RNN and LSTM in long-range context dependencies?
I'll list some bullet points of the main innovations introduced by transformers , followed by bullet points of the main characteristics of the other architectures you mentioned, so we can then ...
- 3,433
17
votes
What is the intuition behind the dot product attention?
Let's start with a bit of notation and a couple of important clarifications.
$\mathbf{Q}$ refers to the query vectors matrix, $q_i$ being a single query vector associated with a single input word.
$\...
- 3,433
13
votes
Accepted
How is BERT different from the original transformer architecture?
What is a transformer?
The original transformer, proposed in the paper Attention is all you need (2017), is an encoder-decoder-based neural network that is mainly characterized by the use of the so-...
- 33k
12
votes
Accepted
How can Transformers handle arbitrary length input?
Actually, there is usually an upper bound for inputs of transformers, due to the inability of handling long-sequence. Usually, the value is set as 512 or 1024 at current stage.
However, if you are ...
- 181
8
votes
Why does the transformer do better than RNN and LSTM in long-range context dependencies?
Let's start with RNN. A well known problem is vanishin/exploding gradients, which means that the model is biased by most recent inputs in the sequence, or in other words, older inputs have practically ...
- 266
7
votes
Accepted
What exactly are the "parameters" in GPT-3's 175 billion parameters and how are they chosen/generated?
Parameters is a synonym for weights, which is the term most people use for a neural networks parameters (and indeed in my experience it is a term that machine learners will use in general whereas ...
- 4,016
6
votes
Can the decoder in a transformer model be parallelized like the encoder?
Can the decoder in a transformer model be parallelized like the encoder?
The correct answer is: computation in a Transformer decoder can be parallelized during training, but not during actual ...
- 311
6
votes
Accepted
What kind of word embedding is used in the original transformer?
I have found a good answer in this blog post The Transformer: Attention Is All You Need:
we learn a “word embedding” which is a smaller real-valued vector representation of the word that carries some ...
- 405
6
votes
Accepted
In Computer Vision, what is the difference between a transformer and attention?
The original transformer is a feedforward neural network (FFNN)-based architecture that makes use of an attention mechanism. So, this is the difference: an attention mechanism (in particular, a self-...
- 33k
5
votes
Accepted
What is the purpose of Decoder mask (triangular mask) in Transformer?
The Transformer model presented in this tutorial is an auto-regressive Transformer. Which means that prediction of next token only depends on it's previous tokens.
So in order to predict next token, ...
- 220
5
votes
Accepted
What is different in each head of a multi-head attention mechanism?
The reason each head is different is because they each learn a different set of weight matrices $\{ W_i^Q, W_i^K, W_i^V \}$ where $i$ is the index of the head. To clarify, the input to each attention ...
- 1,210
5
votes
Why does GPT-2 Exclude the Transformer Encoder?
GPT-2 is a close copy of the basic transformer architecture.
GPT-2 does not require the encoder part of the original transformer architecture as it is decoder-only, and there are no encoder attention ...
- 619
4
votes
Accepted
Can the decoder in a transformer model be parallelized like the encoder?
Can the decoder in a transformer model be parallelized like the
encoder?
Generally NO:
Your understanding is completely right. In the decoder, the output of each step is fed to the bottom decoder in ...
- 379
4
votes
What is the purpose of Decoder mask (triangular mask) in Transformer?
We give the target input into the transformer decoder while training the model. So it is easy for the model to "peek ahead" and learn what the next word would be. To ensure that this doesn't ...
- 141
4
votes
Accepted
Why does this multiplication of $Q$ and $K$ have a variance of $d_k$, in scaled dot product attention?
In statistics, if $X$ and $Y$ are independent and randomly distributed variables:
$\mathbb{E}[X + Y] = \mathbb{E}[X] + \mathbb{E}[Y] \\
Var(X + Y) = Var(X) + Var(Y) \\
\mathbb{E}[XY] = \mathbb{E}[X]\...
- 1,210
4
votes
Accepted
What is the cost function of a transformer?
I took a look at the Tensor2Tensor's source code implementation, and it seems like the loss function is the cross-entropy between the predicted probability matrix $\|\text{sentence length}\| \times \|\...
- 1,210
4
votes
In layman terms, what does "attention" do in a transformer?
Let's start by stressing out that in the literature unfortunately the term attention is still used widely without any precise consensus around the technical details, the only constant across papers is ...
- 3,433
4
votes
Accepted
What are the major layers in a Vision Transformer?
The Transformer family of architectures is a separate family of NN architectures, different from the CNNs and RNNs.
The main part of the Vision Transformer are the self-attention layers.
The ...
- 1,754
3
votes
Accepted
Any comparison between transformer and RNN+Attention on the same dataset?
If you go through the main introductory paper of the transformer ("Attention is all you need"), you can find the comparison of the model with other state-of-the-art machine translation ...
- 1,663
3
votes
Why would you implement the position-wise feed-forward network of the transformer with convolution layers?
I'm going to post another guess to this question - it won't be a complete answer, but hopefully it'll provide some direction towards finding a more legitimate answer.
The feed-forward networks as ...
- 81
3
votes
Accepted
Transformers: how to get the output (keys and values) of the encoder?
I have read the OpenNMT source code (https://github.com/OpenNMT/OpenNMT-py/blob/cd29c1dbfb35f4a2701ff52a1bf4e5bdcf02802e/onmt/modules/multi_headed_attn.py).
It seems like an extra linear layer learns ...
- 1,210
3
votes
Accepted
How are certain machine learning models able to produce variable-length outputs given variable-length inputs?
In short, repetition with feedback.
You are correct that machine learning (ML) models such as neural networks work with fixed dimensions for input and output. There are a few different ways to work ...
- 23.1k
3
votes
Accepted
Is it realistic to train a transformer-based model (e.g. GPT) in a self-supervised way directly on the Mel spectrogram?
The reason most music-generation models use discrete representations is because the long-term structures of music are very challenging to model. Note that the MIDI data in MAESTRO (used in the two ...
- 194
3
votes
Accepted
Why do the authors of the T5 paper say that the "architectural changes are orthogonal to the experimental factors"?
"Orthogonal" is often used to mean "independent", as in "independent variable which does not correlate with the other variables". I believe this terminology originates ...
- 5,027
3
votes
What is the Intermediate (dense) layer in between attention-output and encoder-output dense layers within transformer block in PyTorch implementation?
Feedforward layer is an important part of the transformer architecture.
Transformer architecture, in addition to the self-attention layer, that aggregates ...
- 1,754
2
votes
Why would you implement the position-wise feed-forward network of the transformer with convolution layers?
1) The math is the exact same, so from an optimization or mathematical perspective there is no difference
2) Here are my guesses to a possible answer.
Habit: People may just call one over the ...
- 2,229
2
votes
How does a transformer leverage the GPU to be trained faster than RNNs?
A recurrent neural network (RNN) depends on the previous hidden state from the previous time step. That is, an RNN is a function of both the data for the sequence at time $t$ and the hidden state from ...
- 4,016
2
votes
Are embeddings in multi-lingual language models comparable across languages?
Embeddings generated by transformers like Bert or XLM-R are fundamentally different from embeddings learned through language models like GloVe or Word2Vec.
The latter are static, i.e. they are just ...
- 3,433
2
votes
What is the purpose of Decoder mask (triangular mask) in Transformer?
the mask is needed to prevent the decoder from "peeking ahead" at ground truth during training, when using its Attention mechanism.
Encoder:
Both runtime or training:
the encoder will ...
- 200
2
votes
How to implement or avoid masking for transformer?
If you're using a library such as Trax which contains great submodules for various Transformers (Skipping, BERT, Vanilla and Reformer) you can use the inbuilt ...
- 706
Only top scored, non community-wiki answers of a minimum length are eligible