Tag Info

Let's start with a bit of notation and a couple of important clarifications. $\mathbf{Q}$ refers to the query vectors matrix, $q_i$ being a single query vector associated with a single input word. $\... • 3,528 8 votes Accepted What exactly are the "parameters" in GPT-3's 175 billion parameters and how are they chosen/generated? Parameters is a synonym for weights, which is the term most people use for a neural networks parameters (and indeed in my experience it is a term that machine learners will use in general whereas ... • 4,055 8 votes Accepted Is there any artificial intelligence that possesses "concentration"? Douglas Hofstadter's CopyCat architecture for solving letter-string analogy problems was deliberately engineered to maintain a semantically-informed notion of 'salience', i.e. given a variety of ... • 7,046 6 votes Is there any artificial intelligence that possesses "concentration"? Concentration, perhaps easier to grasp as "focus" or "attention", has quite some history in AI. This answer mentions CopyCat, and there was work with neural networks in the 80s as well (e.g. from ... • 1,480 6 votes Accepted What kind of word embedding is used in the original transformer? I have found a good answer in this blog post The Transformer: Attention Is All You Need: we learn a “word embedding” which is a smaller real-valued vector representation of the word that carries some ... • 405 6 votes Accepted In Computer Vision, what is the difference between a transformer and attention? The original transformer is a feedforward neural network (FFNN)-based architecture that makes use of an attention mechanism. So, this is the difference: an attention mechanism (in particular, a self-... • 33.8k 5 votes Accepted What is different in each head of a multi-head attention mechanism? The reason each head is different is because they each learn a different set of weight matrices$\{ W_i^Q, W_i^K, W_i^V \}$where$i$is the index of the head. To clarify, the input to each attention ... • 1,230 5 votes Accepted What is the purpose of Decoder mask (triangular mask) in Transformer? The Transformer model presented in this tutorial is an auto-regressive Transformer. Which means that prediction of next token only depends on it's previous tokens. So in order to predict next token, ... • 220 5 votes Why does GPT-2 Exclude the Transformer Encoder? GPT-2 is a close copy of the basic transformer architecture. GPT-2 does not require the encoder part of the original transformer architecture as it is decoder-only, and there are no encoder attention ... • 629 4 votes Accepted Why does this multiplication of$Q$and$K$have a variance of$d_k$, in scaled dot product attention? In statistics, if$X$and$Y$are independent and randomly distributed variables:$\mathbb{E}[X + Y] = \mathbb{E}[X] + \mathbb{E}[Y] \\ Var(X + Y) = Var(X) + Var(Y) \\ \mathbb{E}[XY] = \mathbb{E}[X]\...
• 1,230

We give the target input into the transformer decoder while training the model. So it is easy for the model to "peek ahead" and learn what the next word would be. To ensure that this doesn't ...
Accepted

• 241