38 votes
Accepted

Why does the transformer do better than RNN and LSTM in long-range context dependencies?

I'll list some bullet points of the main innovations introduced by transformers , followed by bullet points of the main characteristics of the other architectures you mentioned, so we can then ...
user avatar
17 votes

What is the intuition behind the dot product attention?

Let's start with a bit of notation and a couple of important clarifications. $\mathbf{Q}$ refers to the query vectors matrix, $q_i$ being a single query vector associated with a single input word. $\...
user avatar
13 votes
Accepted

How is BERT different from the original transformer architecture?

What is a transformer? The original transformer, proposed in the paper Attention is all you need (2017), is an encoder-decoder-based neural network that is mainly characterized by the use of the so-...
user avatar
  • 33k
12 votes
Accepted

How can Transformers handle arbitrary length input?

Actually, there is usually an upper bound for inputs of transformers, due to the inability of handling long-sequence. Usually, the value is set as 512 or 1024 at current stage. However, if you are ...
user avatar
  • 181
8 votes

Why does the transformer do better than RNN and LSTM in long-range context dependencies?

Let's start with RNN. A well known problem is vanishin/exploding gradients, which means that the model is biased by most recent inputs in the sequence, or in other words, older inputs have practically ...
user avatar
  • 266
7 votes
Accepted

What exactly are the "parameters" in GPT-3's 175 billion parameters and how are they chosen/generated?

Parameters is a synonym for weights, which is the term most people use for a neural networks parameters (and indeed in my experience it is a term that machine learners will use in general whereas ...
user avatar
6 votes

Can the decoder in a transformer model be parallelized like the encoder?

Can the decoder in a transformer model be parallelized like the encoder? The correct answer is: computation in a Transformer decoder can be parallelized during training, but not during actual ...
user avatar
6 votes
Accepted

What kind of word embedding is used in the original transformer?

I have found a good answer in this blog post The Transformer: Attention Is All You Need: we learn a “word embedding” which is a smaller real-valued vector representation of the word that carries some ...
user avatar
6 votes
Accepted

In Computer Vision, what is the difference between a transformer and attention?

The original transformer is a feedforward neural network (FFNN)-based architecture that makes use of an attention mechanism. So, this is the difference: an attention mechanism (in particular, a self-...
user avatar
  • 33k
5 votes
Accepted

What is the purpose of Decoder mask (triangular mask) in Transformer?

The Transformer model presented in this tutorial is an auto-regressive Transformer. Which means that prediction of next token only depends on it's previous tokens. So in order to predict next token, ...
user avatar
  • 220
5 votes
Accepted

What is different in each head of a multi-head attention mechanism?

The reason each head is different is because they each learn a different set of weight matrices $\{ W_i^Q, W_i^K, W_i^V \}$ where $i$ is the index of the head. To clarify, the input to each attention ...
user avatar
  • 1,210
5 votes

Why does GPT-2 Exclude the Transformer Encoder?

GPT-2 is a close copy of the basic transformer architecture. GPT-2 does not require the encoder part of the original transformer architecture as it is decoder-only, and there are no encoder attention ...
user avatar
  • 619
4 votes
Accepted

Can the decoder in a transformer model be parallelized like the encoder?

Can the decoder in a transformer model be parallelized like the encoder? Generally NO: Your understanding is completely right. In the decoder, the output of each step is fed to the bottom decoder in ...
user avatar
  • 379
4 votes

What is the purpose of Decoder mask (triangular mask) in Transformer?

We give the target input into the transformer decoder while training the model. So it is easy for the model to "peek ahead" and learn what the next word would be. To ensure that this doesn't ...
user avatar
4 votes
Accepted

Why does this multiplication of $Q$ and $K$ have a variance of $d_k$, in scaled dot product attention?

In statistics, if $X$ and $Y$ are independent and randomly distributed variables: $\mathbb{E}[X + Y] = \mathbb{E}[X] + \mathbb{E}[Y] \\ Var(X + Y) = Var(X) + Var(Y) \\ \mathbb{E}[XY] = \mathbb{E}[X]\...
user avatar
  • 1,210
4 votes
Accepted

What is the cost function of a transformer?

I took a look at the Tensor2Tensor's source code implementation, and it seems like the loss function is the cross-entropy between the predicted probability matrix $\|\text{sentence length}\| \times \|\...
user avatar
  • 1,210
4 votes

In layman terms, what does "attention" do in a transformer?

Let's start by stressing out that in the literature unfortunately the term attention is still used widely without any precise consensus around the technical details, the only constant across papers is ...
user avatar
4 votes
Accepted

What are the major layers in a Vision Transformer?

The Transformer family of architectures is a separate family of NN architectures, different from the CNNs and RNNs. The main part of the Vision Transformer are the self-attention layers. The ...
user avatar
3 votes
Accepted

Any comparison between transformer and RNN+Attention on the same dataset?

If you go through the main introductory paper of the transformer ("Attention is all you need"), you can find the comparison of the model with other state-of-the-art machine translation ...
user avatar
  • 1,663
3 votes

Why would you implement the position-wise feed-forward network of the transformer with convolution layers?

I'm going to post another guess to this question - it won't be a complete answer, but hopefully it'll provide some direction towards finding a more legitimate answer. The feed-forward networks as ...
user avatar
3 votes
Accepted

Transformers: how to get the output (keys and values) of the encoder?

I have read the OpenNMT source code (https://github.com/OpenNMT/OpenNMT-py/blob/cd29c1dbfb35f4a2701ff52a1bf4e5bdcf02802e/onmt/modules/multi_headed_attn.py). It seems like an extra linear layer learns ...
user avatar
  • 1,210
3 votes
Accepted

How are certain machine learning models able to produce variable-length outputs given variable-length inputs?

In short, repetition with feedback. You are correct that machine learning (ML) models such as neural networks work with fixed dimensions for input and output. There are a few different ways to work ...
user avatar
  • 23.1k
3 votes
Accepted

Is it realistic to train a transformer-based model (e.g. GPT) in a self-supervised way directly on the Mel spectrogram?

The reason most music-generation models use discrete representations is because the long-term structures of music are very challenging to model. Note that the MIDI data in MAESTRO (used in the two ...
user avatar
  • 194
3 votes
Accepted

Why do the authors of the T5 paper say that the "architectural changes are orthogonal to the experimental factors"?

"Orthogonal" is often used to mean "independent", as in "independent variable which does not correlate with the other variables". I believe this terminology originates ...
user avatar
  • 5,027
3 votes

What is the Intermediate (dense) layer in between attention-output and encoder-output dense layers within transformer block in PyTorch implementation?

Feedforward layer is an important part of the transformer architecture. Transformer architecture, in addition to the self-attention layer, that aggregates ...
user avatar
2 votes

Why would you implement the position-wise feed-forward network of the transformer with convolution layers?

1) The math is the exact same, so from an optimization or mathematical perspective there is no difference 2) Here are my guesses to a possible answer. Habit: People may just call one over the ...
user avatar
  • 2,229
2 votes

How does a transformer leverage the GPU to be trained faster than RNNs?

A recurrent neural network (RNN) depends on the previous hidden state from the previous time step. That is, an RNN is a function of both the data for the sequence at time $t$ and the hidden state from ...
user avatar
2 votes

Are embeddings in multi-lingual language models comparable across languages?

Embeddings generated by transformers like Bert or XLM-R are fundamentally different from embeddings learned through language models like GloVe or Word2Vec. The latter are static, i.e. they are just ...
user avatar
2 votes

What is the purpose of Decoder mask (triangular mask) in Transformer?

the mask is needed to prevent the decoder from "peeking ahead" at ground truth during training, when using its Attention mechanism. Encoder: Both runtime or training: the encoder will ...
user avatar
  • 200
2 votes

How to implement or avoid masking for transformer?

If you're using a library such as Trax which contains great submodules for various Transformers (Skipping, BERT, Vanilla and Reformer) you can use the inbuilt ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible