30 votes

What is the intuition behind the dot product attention?

Let's start with a bit of notation and a couple of important clarifications. $\mathbf{Q}$ refers to the query vectors matrix, $q_i$ being a single query vector associated with a single input word. $\...
Edoardo Guerriero's user avatar
29 votes
Accepted

How does the (decoder-only) transformer architecture work?

Introduction Large-language models (LLMs) have gained tons of popularity lately with the releases of ChatGPT, GPT-4, Bard, and more. All these LLMs are based on the transformer neural network ...
Robin van Hoorn's user avatar
24 votes
Accepted

What exactly are the "parameters" in GPT-3's 175 billion parameters and how are they chosen/generated?

Parameters is a synonym for weights, which is the term most people use for a neural networks parameters (and indeed in my experience it is a term that machine learners will use in general whereas ...
David's user avatar
  • 4,760
16 votes

Why does GPT-2 Exclude the Transformer Encoder?

GPT-2 is a close copy of the basic transformer architecture. GPT-2 does not require the encoder part of the original transformer architecture as it is decoder-only, and there are no encoder attention ...
Faizy's user avatar
  • 1,114
15 votes
Accepted

What kind of word embedding is used in the original transformer?

I have found a good answer in this blog post The Transformer: Attention Is All You Need: we learn a “word embedding” which is a smaller real-valued vector representation of the word that carries some ...
Bert Gayus's user avatar
12 votes
Accepted

What is the purpose of Decoder mask (triangular mask) in Transformer?

The Transformer model presented in this tutorial is an auto-regressive Transformer. Which means that prediction of next token only depends on it's previous tokens. So in order to predict next token, ...
Leo's user avatar
  • 300
11 votes
Accepted

Why does this multiplication of $Q$ and $K$ have a variance of $d_k$, in scaled dot product attention?

In statistics, if $X$ and $Y$ are independent and randomly distributed variables: $\mathbb{E}[X + Y] = \mathbb{E}[X] + \mathbb{E}[Y] \\ Var(X + Y) = Var(X) + Var(Y) \\ \mathbb{E}[XY] = \mathbb{E}[X]\...
user3667125's user avatar
  • 1,530
9 votes
Accepted

In Computer Vision, what is the difference between a transformer and attention?

The original transformer is a feedforward neural network (FFNN)-based architecture that makes use of an attention mechanism. So, this is the difference: an attention mechanism (in particular, a self-...
nbro's user avatar
  • 40.1k
9 votes
Accepted

Why does a transformer not use an activation function following the multi-head attention layer?

This goes back to the purpose of self-attention. Measure between word-vectors is generally computed through cosine-similarity because in the dimensions word tokens exist, it's highly unlikely for two ...
Avatrin's user avatar
  • 486
8 votes
Accepted

Is there any artificial intelligence that possesses "concentration"?

Douglas Hofstadter's CopyCat architecture for solving letter-string analogy problems was deliberately engineered to maintain a semantically-informed notion of 'salience', i.e. given a variety of ...
NietzscheanAI's user avatar
8 votes
Accepted

What is different in each head of a multi-head attention mechanism?

The reason each head is different is because they each learn a different set of weight matrices $\{ W_i^Q, W_i^K, W_i^V \}$ where $i$ is the index of the head. To clarify, the input to each attention ...
user3667125's user avatar
  • 1,530
8 votes

How to assess if OpenAI's ChatGPT chatbot has a human in the loop?

Until you can prove that OpenAI has HITL in ChatGPT, it is just an idea with no basis. It's not up to us to disprove it, it's up to you to prove it. Let me address your points: You seem to be basing ...
GalacticRaph's user avatar
8 votes
Accepted

Why are biases (typically) not used in attention mechanism?

For certain types of layers, such as transformers and convolutional layers, including a bias term is unnecessary and adds unnecessary overhead to the model. The reason for this is that these layers ...
Marc Dumon's user avatar
8 votes

Can you confirm that the transformer works strictly deterministically and there is no randomness inside or between the attention layers?

Can you confirm that the transformer works strictly deterministically and there is no randomness inside or between the attention layers? Of course, there is no injected randomness in a regular ...
Luca Anzalone's user avatar
7 votes
Accepted

What is the cost function of a transformer?

I took a look at the Tensor2Tensor's source code implementation, and it seems like the loss function is the cross-entropy between the predicted probability matrix $\|\text{sentence length}\| \times \|\...
user3667125's user avatar
  • 1,530
7 votes

Isn't attention mask for BERT model useless?

This is just an implementation issue. One reason is the Huggingface implementation (which is not the original implementation by Google) wants to strictly separate the tokenization from the modeling. ...
Jindřich's user avatar
  • 391
6 votes

What is the purpose of Decoder mask (triangular mask) in Transformer?

the mask is needed to prevent the decoder from "peeking ahead" at ground truth during training, when using its Attention mechanism. Encoder: Both runtime or training: the encoder will ...
Kari's user avatar
  • 270
6 votes

What is the purpose of Decoder mask (triangular mask) in Transformer?

We give the target input into the transformer decoder while training the model. So it is easy for the model to "peek ahead" and learn what the next word would be. To ensure that this doesn't ...
Hari Krishnan's user avatar
6 votes

Is there any artificial intelligence that possesses "concentration"?

Concentration, perhaps easier to grasp as "focus" or "attention", has quite some history in AI. This answer mentions CopyCat, and there was work with neural networks in the 80s as well (e.g. from ...
Eric Platon's user avatar
  • 1,500
6 votes

Why does GPT-2 Exclude the Transformer Encoder?

The cases when we use encoder-decoder architectures are typically when we are mapping one type of sequence to another type of sequence, e.g. translating French to English or in the case of a chatbot ...
Ben's user avatar
  • 81
6 votes

Why are embeddings added, not concatenated?

In high-dimensional spaces, the token embeddings and positional encodings can be thought of as forming two separate subspaces that are approximately orthogonal to each other. This is based on the ...
Abdur Rahman's user avatar
5 votes

What's the difference between content-based attention and dot-product attention?

The Attention is All you Need has this footnote at the passage motivating the introduction of the $1/\sqrt{d_k}$ factor: To illustrate why the dot products get large, assume that the components of $q$...
Kostya's user avatar
  • 2,496
5 votes

What kind of word embedding is used in the original transformer?

No, neither Word2Vec nor GloVe is used as Transformers are a newer class of algorithms. Word2Vec and GloVe are based on static word embeddings while Transformers are based on dynamic word embeddings. ...
Brian O'Donnell's user avatar
5 votes

Can someone help me understand the intuition behind the query, key and value matrices in the transformer architecture?

A very very distant connection can be seen between the self-attention layer and the word2vec model. I think that this might be helpful to in order to gain more intution. Starting from the word2vec ...
pi-tau's user avatar
  • 692
4 votes
Accepted

What is the intuition behind the attention mechanism?

Simply put, the attention mechanism is loosely inspired on well, attention. Consider we are attempting machine translation on the following sentence: "The dog is a Labrador." If you were to ask ...
Archie Shahidullah's user avatar
4 votes

Why does this multiplication of $Q$ and $K$ have a variance of $d_k$, in scaled dot product attention?

Assume that the query embeddings $Q$ and key embeddings $K$ have zero mean and unit std. Then for the variance of the attention score between any query and key we get: $$ \alpha = q_i k_j^T = \sum_{n=...
pi-tau's user avatar
  • 692
4 votes

Why don't people use nonlinear activation functions after projecting the query key value in attention?

It seems like doing this would lead to much-needed nonlinearity, otherwise, we're just doing linear transformations. Attention is broadly defined as a following operation ($\text{softmax}$ is ...
Kostya's user avatar
  • 2,496
4 votes

What is the gradient of an attention unit?

I have written a blog to answer this question, please see https://say-hello2y.github.io/2022-09-07/attention-gradient I use Matrix calculus to solve this question, here I put the final result of the ...
He LongXiang's user avatar
4 votes

Are there any advantages of the local attention against convolutions?

It is true that when using local attention with a window of size 5, the "receptive field" is the same as a CNN with kernel size 5 (or two CNN layers with kernel size 3). However, there is a ...
TimD1's user avatar
  • 161
4 votes

Is there a proper initialization technique for the weight matrices in multi-head attention?

IMO xavier/glorot is the correct way to initialize the $W_Q$ and $W_K$ matrices. In section 3.2.1 of the transformer paper the ...
pi-tau's user avatar
  • 692

Only top scored, non community-wiki answers of a minimum length are eligible