Skip to main content
Share Your Experience: Take the 2024 Developer Survey
62 votes
Accepted

Why does the transformer do better than RNN and LSTM in long-range context dependencies?

I'll list some bullet points of the main innovations introduced by transformers , followed by bullet points of the main characteristics of the other architectures you mentioned, so we can then ...
Edoardo Guerriero's user avatar
36 votes
Accepted

How does the (decoder-only) transformer architecture work?

Introduction Large-language models (LLMs) have gained tons of popularity lately with the releases of ChatGPT, GPT-4, Bard, and more. All these LLMs are based on the transformer neural network ...
Robin van Hoorn's user avatar
31 votes

What is the intuition behind the dot product attention?

Let's start with a bit of notation and a couple of important clarifications. $\mathbf{Q}$ refers to the query vectors matrix, $q_i$ being a single query vector associated with a single input word. $\...
Edoardo Guerriero's user avatar
31 votes
Accepted

How can Transformers handle arbitrary length input?

Actually, there is usually an upper bound for inputs of transformers, due to the inability of handling long-sequence. Usually, the value is set as 512 or 1024 at current stage. However, if you are ...
tsu's user avatar
  • 471
30 votes
Accepted

How is BERT different from the original transformer architecture?

What is a transformer? The original transformer, proposed in the paper Attention is all you need (2017), is an encoder-decoder-based neural network that is mainly characterized by the use of the so-...
nbro's user avatar
  • 40.8k
24 votes
Accepted

What exactly are the "parameters" in GPT-3's 175 billion parameters and how are they chosen/generated?

Parameters is a synonym for weights, which is the term most people use for a neural networks parameters (and indeed in my experience it is a term that machine learners will use in general whereas ...
David's user avatar
  • 4,910
19 votes
Accepted

Can the decoder in a transformer model be parallelized like the encoder?

Can the decoder in a transformer model be parallelized like the encoder? Generally NO: Your understanding is completely right. In the decoder, the output of each step is fed to the bottom decoder in ...
HLeb's user avatar
  • 579
18 votes
Accepted

What kind of word embedding is used in the original transformer?

I have found a good answer in this blog post The Transformer: Attention Is All You Need: we learn a “word embedding” which is a smaller real-valued vector representation of the word that carries some ...
Bert Gayus's user avatar
16 votes

Why does GPT-2 Exclude the Transformer Encoder?

GPT-2 is a close copy of the basic transformer architecture. GPT-2 does not require the encoder part of the original transformer architecture as it is decoder-only, and there are no encoder attention ...
Faizy's user avatar
  • 1,104
12 votes
Accepted

What is the purpose of Decoder mask (triangular mask) in Transformer?

The Transformer model presented in this tutorial is an auto-regressive Transformer. Which means that prediction of next token only depends on it's previous tokens. So in order to predict next token, ...
Leo's user avatar
  • 425
11 votes
Accepted

Why does this multiplication of $Q$ and $K$ have a variance of $d_k$, in scaled dot product attention?

In statistics, if $X$ and $Y$ are independent and randomly distributed variables: $\mathbb{E}[X + Y] = \mathbb{E}[X] + \mathbb{E}[Y] \\ Var(X + Y) = Var(X) + Var(Y) \\ \mathbb{E}[XY] = \mathbb{E}[X]\...
user3667125's user avatar
  • 1,570
10 votes

Why does the transformer do better than RNN and LSTM in long-range context dependencies?

Let's start with RNN. A well known problem is vanishin/exploding gradients, which means that the model is biased by most recent inputs in the sequence, or in other words, older inputs have practically ...
olix20's user avatar
  • 286
10 votes
Accepted

In Computer Vision, what is the difference between a transformer and attention?

The original transformer is a feedforward neural network (FFNN)-based architecture that makes use of an attention mechanism. So, this is the difference: an attention mechanism (in particular, a self-...
nbro's user avatar
  • 40.8k
9 votes

Can the decoder in a transformer model be parallelized like the encoder?

Can the decoder in a transformer model be parallelized like the encoder? The correct answer is: computation in a Transformer decoder can be parallelized during training, but not during actual ...
Mathias Müller's user avatar
9 votes
Accepted

What is different in each head of a multi-head attention mechanism?

The reason each head is different is because they each learn a different set of weight matrices $\{ W_i^Q, W_i^K, W_i^V \}$ where $i$ is the index of the head. To clarify, the input to each attention ...
user3667125's user avatar
  • 1,570
9 votes
Accepted

Why does a transformer not use an activation function following the multi-head attention layer?

This goes back to the purpose of self-attention. Measure between word-vectors is generally computed through cosine-similarity because in the dimensions word tokens exist, it's highly unlikely for two ...
Avatrin's user avatar
  • 496
8 votes

How to assess if OpenAI's ChatGPT chatbot has a human in the loop?

Until you can prove that OpenAI has HITL in ChatGPT, it is just an idea with no basis. It's not up to us to disprove it, it's up to you to prove it. Let me address your points: You seem to be basing ...
GalacticRaph's user avatar
8 votes
Accepted

Why are biases (typically) not used in attention mechanism?

For certain types of layers, such as transformers and convolutional layers, including a bias term is unnecessary and adds unnecessary overhead to the model. The reason for this is that these layers ...
Marc Dumon's user avatar
8 votes

Can you confirm that the transformer works strictly deterministically and there is no randomness inside or between the attention layers?

Can you confirm that the transformer works strictly deterministically and there is no randomness inside or between the attention layers? Of course, there is no injected randomness in a regular ...
Luca Anzalone's user avatar
8 votes
Accepted

While fine-tuning a decoder only LLM like LLaMA on chat dataset, what kind of padding should one use?

I got an answer to this question, probably a correct explanation. In decoder-only model architectures, the output of the model is a continuation of the model input. For example, input: I love apple [...
尹雅博's user avatar
7 votes

How can Transformers handle arbitrary length input?

The accepted answer is wonderful; this answer provides an alternative approach for dealing with variable length inputs. More specifically, what might be done when the input is longer than the maximum ...
David Hoelzer's user avatar
7 votes
Accepted

What is the cost function of a transformer?

I took a look at the Tensor2Tensor's source code implementation, and it seems like the loss function is the cross-entropy between the predicted probability matrix $\|\text{sentence length}\| \times \|\...
user3667125's user avatar
  • 1,570
7 votes
Accepted

Transformers: How to use the target mask properly?

The main issue during training is that you haven't right-shifted the input of the decoder, which is probably why you set the diagonals of mask to -inf (when it ...
user3667125's user avatar
  • 1,570
7 votes

Why are embeddings added, not concatenated?

In high-dimensional spaces, the token embeddings and positional encodings can be thought of as forming two separate subspaces that are approximately orthogonal to each other. This is based on the ...
Abdur Rahman's user avatar
7 votes
Accepted

Which situation will helpful using encoder or decoder or both in transformer model?

The original transformer paper presents the transformer as a model consisting of both encoder and decoder. However, many times you will see (or hear) people describing their model as a "...
pi-tau's user avatar
  • 805
7 votes
Accepted

How can Transformers handle random sequences?

The stage of setting the available token encodings when training LLMs is very early on. It is before, and separate to the token prediction training (the core training process for the models). It is ...
Neil Slater's user avatar
  • 32.4k
6 votes

What is the purpose of Decoder mask (triangular mask) in Transformer?

We give the target input into the transformer decoder while training the model. So it is easy for the model to "peek ahead" and learn what the next word would be. To ensure that this doesn't ...
Hari Krishnan's user avatar
6 votes

What is the purpose of Decoder mask (triangular mask) in Transformer?

the mask is needed to prevent the decoder from "peeking ahead" at ground truth during training, when using its Attention mechanism. Encoder: Both runtime or training: the encoder will ...
Kari's user avatar
  • 270
6 votes
Accepted

What is the difference between the positional encoding techniques of the Transformer and GPT?

The purpose of introduction of positional encoding is to insert a notion of location of a given token in the sequence. Without it, due to the permutation equivariance (symmetry under the token ...
spiridon_the_sun_rotator's user avatar
6 votes

Why does GPT-2 Exclude the Transformer Encoder?

The cases when we use encoder-decoder architectures are typically when we are mapping one type of sequence to another type of sequence, e.g. translating French to English or in the case of a chatbot ...
Ben's user avatar
  • 81

Only top scored, non community-wiki answers of a minimum length are eligible