10

Let's start with a bit of notation and a couple of important clarification. Q refers to the query vectors matrix, $q_i$ being a single query vector associated with a single input word. V refers to the values vectors matrix, $v_i$ being a single value vector associated with a single input word. K refers to the keys vectors matrix, $k_i$ being a single key ...


8

Douglas Hofstadter's CopyCat architecture for solving letter-string analogy problems was deliberately engineered to maintain a semantically-informed notion of 'salience', i.e. given a variety of competing possibilities, tend to maintain interest in the one that is most compelling. Although the salience value of (part of) a solution is ultimately represented ...


7

I think there are two parts to answering this question. First, about the specific paper that has been mentioned. The paper's title is hyperbolic, and probably written that way to get more people to read it. The paper itself does not make the claim that attention-based networks will supplant existing recurrent network architectures. Instead, it makes a more ...


6

Concentration, perhaps easier to grasp as "focus" or "attention", has quite some history in AI. This answer mentions CopyCat, and there was work with neural networks in the 80s as well (e.g. from Fukushima, creator of the Neocognitron). More recently, attention in neural networks is gaining momentum. The mechanisms are applied to learning in deep neural ...


5

Parameters is a synonym for weights, which is the term most people use for a neural networks parameters (and indeed in my experience it is a term that machine learners will use in general whereas parameters is more often found in statistics literature). Batch size, learning rate etc. are hyper-parameters which basically means they are user specified, whereas ...


4

We give the target input into the transformer decoder while training the model. So it is easy for the model to "peek ahead" and learn what the next word would be. To ensure that this doesn't happen we apply an additive mask after the dot product between Query and Key. In the original paper "Attention is all you need", the triangular ...


4

The Transformer model presented in this tutorial is an auto-regressive Transformer. Which means that prediction of next token only depends on it's previous tokens. So in order to predict next token, you have to make sure that only previous token are attended. (If not, this would be a cheating because model already knows whats next). So attention mask would ...


4

I took a look at the Tensor2Tensor's source code implementation as per @nbro's suggestion, and it seems like the loss function is the cross entropy between the predicted ($\|sentence length\|$ x $\|vocab\|$) probability matrix (right before taking the argmax to find the token to output), and the $\|sentence length\|$-length vector of token IDs as the true ...


4

GPT-2 is a close copy of the basic transformer architecture. GPT-2 does not require the encoder part of the original transformer architecture as it is decoder-only, and there are no encoder attention blocks, so the decoder is equivalent to the encoder, except for the MASKING in the multi-head attention block, the decoder is only allowed to glean information ...


3

I have read the OpenNMT source code (https://github.com/OpenNMT/OpenNMT-py/blob/cd29c1dbfb35f4a2701ff52a1bf4e5bdcf02802e/onmt/modules/multi_headed_attn.py). It seems like an extra linear layer learns the weights $W^{key}$ and $W^{value}$ (plus biases), so to get the output (keys and values), you multiply the output of the encoder's final add + norm layer by $...


3

The Attention is All you Need has this footnote at the passage motivating the introduction of the $1/\sqrt{d_k}$ factor: To illustrate why the dot products get large, assume that the components of $q$ and $k$ are independent random variables with mean 0 and variance 1. Then their dot product, $q \cdot k = \sum^{d_k}_{i=1}q_ik_i$ has mean 0 and variance $d_k$...


3

If you go through the main introductory paper of the transformer ("Attention is all you need"), you can find the comparison of the model with other state-of-the-art machine translation method: For example, Deep-Att + PosUnk is a method that has utilized RNN and attention for the translation task. As you can see, the training cost for the ...


3

In statistics, if $X$ and $Y$ are independent and randomly distributed variables: $\mathbb{E}[X + Y] = \mathbb{E}[X] + \mathbb{E}[Y] \\ Var(X + Y) = Var(X) + Var(Y) \\ \mathbb{E}[XY] = \mathbb{E}[X]\mathbb{E}[Y] \\ Var(XY) = (Var(X) + \mathbb{E}[X]^2)(Var(Y) + \mathbb{E}[Y]^2) - \mathbb{E}[X]^2\mathbb{E}[Y]^2$ Let $Q$ and $K$ be random $d_k$ x $d_k$ matrices,...


2

There's plenty, but keep in mind that these articles do not describe the same approach. They simply have attention shifting automation as part of their approaches and therefore must detect a need for shift and execute it in a way that improves speed, accuracy, reliability or some combination of them. There is no one dominant attention approach and probably ...


2

The additive attention method that the researchers are comparing to corresponds to a neural network with 3 layers (it is not actually straight addition). Computing this will involve one multiplication of the input vector by a matrix, then by another matrix, and then the computation of something like a softmax. Smart implementation of a dot-product will not ...


2

Simply put, the attention mechanism is loosely inspired on well, attention. Consider we are attempting machine translation on the following sentence: "The dog is a Labrador." If you were to ask someone to pick out the key words of the sentence, i.e. which ones encode the most meaning, they would likely say "dog" and "Labrador." Articles like "the" and "a" ...


2

There are a few more books that were published after 2016 that cover some of the topics you are interested in. I've not read any of them, so I don't really know whether they are good or not, but I try to summarise if they cover some of the topics you may be interested in. Deep Learning with Python (2017), by Francois Chollet (author of the initial Keras ...


2

I recommend Introduction to Deep Learning by Eugene Charniak ISBN 978-0-262-03951-2 (MIT 2018). It mentions GAN & LSTM & Attention (all three occurs in the index). But read also Pitrat's last book: Artificial Beings: The Conscience of a Conscious Machine - it does cover machine learning (but not in the "deep learning" sense) but was ...


2

The reason each head is different is because they each learn a different set of weight matrices $\{ W_i^Q, W_i^K, W_i^V \}$ where $i$ is the index of the head. To clarify, the input to each attention head is the same. For attention head $i$: $Q_i(x) = x W_i^Q \\ K_i(x) = x W_i^K \\ V_i(x) = x W_i^V \\ attention_i(x) = softmax \left(\frac{Q_i(x) K_i(x)^T}{\...


2

CNNs work by applying filters over the entire image. The same filter is applied at every pixel in the image. That is, the same weights are used at every pixel. Note, when I say "at every pixel" this means across the spatial dimension HxW of the image. You can also have attention in the channel dimension. See for example Squeeze and Excitation: ...


2

This is Jensen's inequality at work. First of all, note that the first line can be rewritten as an expectation $$\sum_{s} p(s \mid \mathbf{a}) \log p(\mathbf{y} \mid s, \mathbf{a}) = \mathbb{E}_{p(s|a)}[\log p(\mathbf{y} \mid s, \mathbf{a})]$$ Then Jensen's inequality gives (Note that a log function is a concave function so gives the opposite inequality to ...


2

The original transformer is a feedforward neural network (FFNN)-based architecture that makes use of an attention mechanism. So, this is the difference: an attention mechanism (in particular, a self-attention operation) is used by the transformer, which is not just this attention mechanism, but it's an encoder-decoder architecture, which makes use of other ...


2

Read the paper. It tells you. (page 3) Bit memory. Similar to the task proposed by Miconi et al. (2018), we consider a bit memory task where the model is shown 5 bitstrings each of length 1000. Afterwards, the model is shown a masked version of one of the bitstrings, where each bit is masked with probability 0.5, and the model is tasked with producing the ...


1

I have found a good answer in this blog post The Transformer: Attention Is All You Need: we learn a “word embedding” which is a smaller real-valued vector representation of the word that carries some information about the word. We can do this using nn.Embedding in Pytorch, or, more generally speaking, by multiplying our one-hot vector with a learned weight ...


1

No, neither Word2Vec nor GloVe is used as Transformers are a newer class of algorithms. Word2Vec and GloVe are based on static word embeddings while Transformers are based on dynamic word embeddings. The embeddings are trained from scratch.


1

Weights are not normally shared across Transformer layers in vanilla Transformers. However, there has been research done in testing out sharing weights, and sometimes they improve the scores. Here are some examples: ALBERT is an improvement on BERT (so only uses the encoding side, no decoder), and shows that sharing the attention weights only $\left\{ W_i^Q, ...


1

I'll use notation from the paper you cited, and any other readers should refer to the paper (widely available) for definitions of notation. The utility of using $W^Q$ and $W^K$, rather than $W$, lies in the fact that they allow us to add fewer parameters to our architecture. $W$ has dimension $d_{model} \times d_{model}$, which means that we are adding $d_{...


1

The masking should be applied to all Decoder blocks, otherwise in some blocks, past words can attend to future words, which would be cheating during training. This is reflected in The Annotated Transformer as well. Notice that in the Decoder class, the forward function applies the same mask to each layer of the decoder: class Decoder(nn.Module): "...


1

I found the answer by reading the paper referenced by that section, Using the output embedding to improve language models Based on this observation, we propose threeway weight tying (TWWT), where the input embedding of the decoder, the output embedding of the decoder and the input embedding of the encoder are all tied. The single source/target vocabulary of ...


1

The authors of the original paper don't provide an explanation, but I suspect it's a combination of: popular recognizable branding (cf. BERT, DALL-E, Watson etc) similarity to [sequence] transduction / translation / transformations generally


Only top voted, non community-wiki answers of a minimum length are eligible