18 votes

What is the intuition behind the dot product attention?

Let's start with a bit of notation and a couple of important clarifications. $\mathbf{Q}$ refers to the query vectors matrix, $q_i$ being a single query vector associated with a single input word. $\...
user avatar
8 votes
Accepted

What exactly are the "parameters" in GPT-3's 175 billion parameters and how are they chosen/generated?

Parameters is a synonym for weights, which is the term most people use for a neural networks parameters (and indeed in my experience it is a term that machine learners will use in general whereas ...
user avatar
8 votes
Accepted

Is there any artificial intelligence that possesses "concentration"?

Douglas Hofstadter's CopyCat architecture for solving letter-string analogy problems was deliberately engineered to maintain a semantically-informed notion of 'salience', i.e. given a variety of ...
user avatar
6 votes

Is there any artificial intelligence that possesses "concentration"?

Concentration, perhaps easier to grasp as "focus" or "attention", has quite some history in AI. This answer mentions CopyCat, and there was work with neural networks in the 80s as well (e.g. from ...
user avatar
  • 1,480
6 votes
Accepted

What kind of word embedding is used in the original transformer?

I have found a good answer in this blog post The Transformer: Attention Is All You Need: we learn a “word embedding” which is a smaller real-valued vector representation of the word that carries some ...
user avatar
6 votes
Accepted

In Computer Vision, what is the difference between a transformer and attention?

The original transformer is a feedforward neural network (FFNN)-based architecture that makes use of an attention mechanism. So, this is the difference: an attention mechanism (in particular, a self-...
user avatar
  • 33.8k
5 votes
Accepted

What is different in each head of a multi-head attention mechanism?

The reason each head is different is because they each learn a different set of weight matrices $\{ W_i^Q, W_i^K, W_i^V \}$ where $i$ is the index of the head. To clarify, the input to each attention ...
user avatar
  • 1,230
5 votes
Accepted

What is the purpose of Decoder mask (triangular mask) in Transformer?

The Transformer model presented in this tutorial is an auto-regressive Transformer. Which means that prediction of next token only depends on it's previous tokens. So in order to predict next token, ...
user avatar
  • 220
5 votes

Why does GPT-2 Exclude the Transformer Encoder?

GPT-2 is a close copy of the basic transformer architecture. GPT-2 does not require the encoder part of the original transformer architecture as it is decoder-only, and there are no encoder attention ...
user avatar
  • 629
4 votes
Accepted

Why does this multiplication of $Q$ and $K$ have a variance of $d_k$, in scaled dot product attention?

In statistics, if $X$ and $Y$ are independent and randomly distributed variables: $\mathbb{E}[X + Y] = \mathbb{E}[X] + \mathbb{E}[Y] \\ Var(X + Y) = Var(X) + Var(Y) \\ \mathbb{E}[XY] = \mathbb{E}[X]\...
user avatar
  • 1,230
4 votes

What is the purpose of Decoder mask (triangular mask) in Transformer?

We give the target input into the transformer decoder while training the model. So it is easy for the model to "peek ahead" and learn what the next word would be. To ensure that this doesn't ...
user avatar
4 votes
Accepted

What is the cost function of a transformer?

I took a look at the Tensor2Tensor's source code implementation, and it seems like the loss function is the cross-entropy between the predicted probability matrix $\|\text{sentence length}\| \times \|\...
user avatar
  • 1,230
4 votes

What's the difference between content-based attention and dot-product attention?

The Attention is All you Need has this footnote at the passage motivating the introduction of the $1/\sqrt{d_k}$ factor: To illustrate why the dot products get large, assume that the components of $q$...
user avatar
  • 1,813
4 votes

In layman terms, what does "attention" do in a transformer?

Let's start by stressing out that in the literature unfortunately the term attention is still used widely without any precise consensus around the technical details, the only constant across papers is ...
user avatar
3 votes

What is the difference between Attention Gate and CNN filters?

CNNs work by applying filters over the entire image. The same filter is applied at every pixel in the image. That is, the same weights are used at every pixel. Note, when I say "at every pixel&...
user avatar
3 votes
Accepted

Transformers: how to get the output (keys and values) of the encoder?

I have read the OpenNMT source code (https://github.com/OpenNMT/OpenNMT-py/blob/cd29c1dbfb35f4a2701ff52a1bf4e5bdcf02802e/onmt/modules/multi_headed_attn.py). It seems like an extra linear layer learns ...
user avatar
  • 1,230
3 votes
Accepted

Any comparison between transformer and RNN+Attention on the same dataset?

If you go through the main introductory paper of the transformer ("Attention is all you need"), you can find the comparison of the model with other state-of-the-art machine translation ...
user avatar
  • 1,663
3 votes
Accepted

What is the intuition behind the attention mechanism?

Simply put, the attention mechanism is loosely inspired on well, attention. Consider we are attempting machine translation on the following sentence: "The dog is a Labrador." If you were to ask ...
user avatar
3 votes

Isn't attention mask for BERT model useless?

This is just an implementation issue. One reason is the Huggingface implementation (which is not the original implementation by Google) wants to strictly separate the tokenization from the modeling. ...
user avatar
  • 141
2 votes

Why is dot product attention faster than additive attention?

The additive attention method that the researchers are comparing to corresponds to a neural network with 3 layers (it is not actually straight addition). Computing this will involve one multiplication ...
user avatar
2 votes

How does a transformer leverage the GPU to be trained faster than RNNs?

A recurrent neural network (RNN) depends on the previous hidden state from the previous time step. That is, an RNN is a function of both the data for the sequence at time $t$ and the hidden state from ...
user avatar
2 votes

Why are "Transformers" called this way?

The authors of the original paper don't provide an explanation, but I suspect it's a combination of: popular recognizable branding (cf. BERT, DALL-E, Watson etc) similarity to [sequence] transduction ...
user avatar
2 votes

What is the purpose of Decoder mask (triangular mask) in Transformer?

the mask is needed to prevent the decoder from "peeking ahead" at ground truth during training, when using its Attention mechanism. Encoder: Both runtime or training: the encoder will ...
user avatar
  • 200
2 votes

Recent deep learning textbook (i.e. covering at least GANs, LSTM and transformers and attention)

I recommend Introduction to Deep Learning by Eugene Charniak ISBN 978-0-262-03951-2 (MIT 2018). It mentions GAN & LSTM & ...
user avatar
2 votes
Accepted

Recent deep learning textbook (i.e. covering at least GANs, LSTM and transformers and attention)

There are a few more books that were published after 2016 that cover some of the topics you are interested in. I've not read any of them, so I don't really know whether they are good or not, but I try ...
user avatar
  • 33.8k
2 votes

What kind of word embedding is used in the original transformer?

No, neither Word2Vec nor GloVe is used as Transformers are a newer class of algorithms. Word2Vec and GloVe are based on static word embeddings while Transformers are based on dynamic word embeddings. ...
user avatar
2 votes
Accepted

Do RNNs/LSTMs really need to be sequential?

You are talking about model parallelism. But, that's not the reason RNNs/LSTMs are not in vogue. Imagine your ability to read the first line of a page and going on reading and still making connections ...
user avatar
2 votes
Accepted

How is the variational lower bound for hard attention derived in Show, Attend and Tell

This is Jensen's inequality at work. First of all, note that the first line can be rewritten as an expectation $$\sum_{s} p(s \mid \mathbf{a}) \log p(\mathbf{y} \mid s, \mathbf{a}) = \mathbb{E}_{p(s|a)...
user avatar
  • 241
2 votes
Accepted

What is the bit memory task?

Read the paper. It tells you. (page 3) Bit memory. Similar to the task proposed by Miconi et al. (2018), we consider a bit memory task where the model is shown 5 bitstrings each of length 1000. ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible