# Tag Info

9

Let's start with a bit of notation and a couple of important clarification. Q refers to the query vectors matrix, $q_i$ being a single query vector associated with a single input word. V refers to the values vectors matrix, $v_i$ being a single value vector associated with a single input word. K refers to the keys vectors matrix, $k_i$ being a single key ...

7

Douglas Hofstadter's CopyCat architecture for solving letter-string analogy problems was deliberately engineered to maintain a semantically-informed notion of 'salience', i.e. given a variety of competing possibilities, tend to maintain interest in the one that is most compelling. Although the salience value of (part of) a solution is ultimately represented ...

7

I think there are two parts to answering this question. First, about the specific paper that has been mentioned. The paper's title is hyperbolic, and probably written that way to get more people to read it. The paper itself does not make the claim that attention-based networks will supplant existing recurrent network architectures. Instead, it makes a more ...

5

Concentration, perhaps easier to grasp as "focus" or "attention", has quite some history in AI. This answer mentions CopyCat, and there was work with neural networks in the 80s as well (e.g. from Fukushima, creator of the Neocognitron). More recently, attention in neural networks is gaining momentum. The mechanisms are applied to learning in deep neural ...

4

Parameters is a synonym for weights, which is the term most people use for a neural networks parameters (and indeed in my experience it is a term that machine learners will use in general whereas parameters is more often found in statistics literature). Batch size, learning rate etc. are hyper-parameters which basically means they are user specified, whereas ...

4

The Transformer model presented in this tutorial is an auto-regressive Transformer. Which means that prediction of next token only depends on it's previous tokens. So in order to predict next token, you have to make sure that only previous token are attended. (If not, this would be a cheating because model already knows whats next). So attention mask would ...

3

GPT-2 is a close copy of the basic transformer architecture. GPT-2 does not require the encoder part of the original transformer architecture as it is decoder-only, and there are no encoder attention blocks, so the decoder is equivalent to the encoder, except for the MASKING in the multi-head attention block, the decoder is only allowed to glean information ...

3

We give the target input into the transformer decoder while training the model. So it is easy for the model to "peek ahead" and learn what the next word would be. To ensure that this doesn't happen we apply an additive mask after the dot product between Query and Key. In the original paper "Attention is all you need", the triangular ...

3

I took a look at the Tensor2Tensor's source code implementation as per @nbro's suggestion, and it seems like the loss function is the cross entropy between the predicted ($\|sentence length\|$ x $\|vocab\|$) probability matrix (right before taking the argmax to find the token to output), and the $\|sentence length\|$-length vector of token IDs as the true ...

2

There's plenty, but keep in mind that these articles do not describe the same approach. They simply have attention shifting automation as part of their approaches and therefore must detect a need for shift and execute it in a way that improves speed, accuracy, reliability or some combination of them. There is no one dominant attention approach and probably ...

2

The additive attention method that the researchers are comparing to corresponds to a neural network with 3 layers (it is not actually straight addition). Computing this will involve one multiplication of the input vector by a matrix, then by another matrix, and then the computation of something like a softmax. Smart implementation of a dot-product will not ...

2

Simply put, the attention mechanism is loosely inspired on well, attention. Consider we are attempting machine translation on the following sentence: "The dog is a Labrador." If you were to ask someone to pick out the key words of the sentence, i.e. which ones encode the most meaning, they would likely say "dog" and "Labrador." Articles like "the" and "a" ...

2

In statistics, if $X$ and $Y$ are independent and randomly distributed variables: $\mathbb{E}[X + Y] = \mathbb{E}[X] + \mathbb{E}[Y] \\ Var(X + Y) = Var(X) + Var(Y) \\ \mathbb{E}[XY] = \mathbb{E}[X]\mathbb{E}[Y] \\ Var(XY) = (Var(X) + \mathbb{E}[X]^2)(Var(Y) + \mathbb{E}[Y]^2) - \mathbb{E}[X]^2\mathbb{E}[Y]^2$ Let $Q$ and $K$ be random $d_k$ x $d_k$ matrices,...

2

I have read the OpenNMT source code (https://github.com/OpenNMT/OpenNMT-py/blob/cd29c1dbfb35f4a2701ff52a1bf4e5bdcf02802e/onmt/modules/multi_headed_attn.py). It seems like an extra linear layer learns the weights $W^{key}$ and $W^{value}$ (plus biases), so to get the output (keys and values), you multiply the output of the encoder's final add + norm layer by $... 2 The Attention is All you Need has this footnote at the passage motivating the introduction of the$1/\sqrt{d_k}$factor: To illustrate why the dot products get large, assume that the components of$q$and$k$are independent random variables with mean 0 and variance 1. Then their dot product,$q \cdot k = \sum^{d_k}_{i=1}q_ik_i$has mean 0 and variance$d_k$... 2 The reason each head is different is because they each learn a different set of weight matrices$\{ W_i^Q, W_i^K, W_i^V \}$where$i$is the index of the head. To clarify, the input to each attention head is the same. For attention head$i$:$Q_i(x) = x W_i^Q \\ K_i(x) = x W_i^K \\ V_i(x) = x W_i^V \\ attention_i(x) = softmax \left(\frac{Q_i(x) K_i(x)^T}{\...

2

If you go through the main introductory paper of the transformer ("Attention is all you need"), you can find the comparison of the model with other state-of-the-art machine translation method: For example, Deep-Att + PosUnk is a method that has utilized RNN and attention for the translation task. As you can see, the training cost for the ...

1

You are talking about model parallelism. But, that's not the reason RNNs/LSTMs are not in vogue. Imagine your ability to read the first line of a page and going on reading and still making connections to the first line until the end of the page. Can RNNs/LSTMs do that? No. Can Attention (i.e. Transformers) do it? Yes. The reason is simple Attention is ...

1

I have found a good answer in this blog post The Transformer: Attention Is All You Need: we learn a “word embedding” which is a smaller real-valued vector representation of the word that carries some information about the word. We can do this using nn.Embedding in Pytorch, or, more generally speaking, by multiplying our one-hot vector with a learned weight ...

1

No, neither Word2Vec nor GloVe is used as Transformers are a newer class of algorithms. Word2Vec and GloVe are based on static word embeddings while Transformers are based on dynamic word embeddings. The embeddings are trained from scratch.

1

CNNs work by applying filters over the entire image. The same filter is applied at every pixel in the image. That is, the same weights are used at every pixel. Note, when I say "at every pixel" this means across the spatial dimension HxW of the image. You can also have attention in the channel dimension. See for example Squeeze and Excitation: ...

1

There are a few more books that were published after 2016 that cover some of the topics you are interested in. I've not read any of them, so I don't really know whether they are good or not, but I try to summarise if they cover some of the topics you may be interested in. Deep Learning with Python (2017), by Francois Chollet (author of the initial Keras ...

1

I recommend Introduction to Deep Learning by Eugene Charniak ISBN 978-0-262-03951-2 (MIT 2018). It mentions GAN & LSTM & Attention (all three occurs in the index). But read also Pitrat's last book: Artificial Beings: The Conscience of a Conscious Machine - it does cover machine learning (but not in the "deep learning" sense) but was ...

1

Weights are not normally shared across Transformer layers in vanilla Transformers. However, there has been research done in testing out sharing weights, and sometimes they improve the scores. Here are some examples: ALBERT is an improvement on BERT (so only uses the encoding side, no decoder), and shows that sharing the attention weights only \$\left\{ W_i^Q, ...

1

The masking should be applied to all Decoder blocks, otherwise in some blocks, past words can attend to future words, which would be cheating during training. This is reflected in The Annotated Transformer as well. Notice that in the Decoder class, the forward function applies the same mask to each layer of the decoder: class Decoder(nn.Module): "...

1

I found the answer by reading the paper referenced by that section, Using the output embedding to improve language models Based on this observation, we propose threeway weight tying (TWWT), where the input embedding of the decoder, the output embedding of the decoder and the input embedding of the encoder are all tied. The single source/target vocabulary of ...

1

The authors of the original paper don't provide an explanation, but I suspect it's a combination of: popular recognizable branding (cf. BERT, DALL-E, Watson etc) similarity to [sequence] transduction / translation / transformations generally

1

I wouldn't say accuracy in the next word prediction is a good global metric. It would depend on the length of sentences. It's always difficult to predict the first word, cos you don't have any context. And having at least one word in the context it's easier. As long as your error rate is averaged among all predicted words, the accuracy could be higher if the ...

1

The problem is not that RNN flavours such as LSTMs are incapable of keeping track of the "important" parts of the input. They also do not have much trouble recognizing commas in different places. To prove this point, I recommend reading Andrej Karpathy's excelllent write-up about the behaviour of individual RNN "neurons". Addressing specifically this ...

1

Answer to Q1) If sampling for next token do you need to apply mask during inference? Yes you do! The models ability to transfer information across positions was trained in this manner, and changing it up will have unpredictable consequences. Let my try to give an example: Tokens: 1:sally, 2:sold, 3:seashells, 4:on, 5:the, 6:____ In the above you are ...

Only top voted, non community-wiki answers of a minimum length are eligible