20
I'll list some bullet points of the main innovations introduced by transformers , followed by bullet points of the main characteristics of the other architectures you mentioned, so we can then compared them.
Transformers
Transformes (Attention is all you need) were introduced in the context of machine translation with the purpose to avoid recursion in order ...
9
Let's start with a bit of notation and a couple of important clarification.
Q refers to the query vectors matrix, $q_i$ being a single query vector associated with a single input word.
V refers to the values vectors matrix, $v_i$ being a single value vector associated with a single input word.
K refers to the keys vectors matrix, $k_i$ being a single key ...
7
Actually, there is usually an upper bound for inputs of transformers, due to the inability of handling long-sequence. Usually, the value is set as 512 or 1024 at current stage.
However, if you are asking handling the various input size, adding padding token such as [PAD] in BERT model is a common solution. The position of [PAD] token could be masked in self-...
7
I think there are two parts to answering this question.
First, about the specific paper that has been mentioned.
The paper's title is hyperbolic, and probably written that way to get more people to read it. The paper itself does not make the claim that attention-based networks will supplant existing recurrent network architectures. Instead, it makes a more ...
6
Can the decoder in a transformer model be parallelized like the encoder?
The correct answer is: computation in a Transformer decoder can be parallelized during training, but not during actual translation (or, in a wider sense, generating output sequences for new input sequences during a testing phase).
What exactly is parallelized?
Also, it's worth ...
4
Parameters is a synonym for weights, which is the term most people use for a neural networks parameters (and indeed in my experience it is a term that machine learners will use in general whereas parameters is more often found in statistics literature). Batch size, learning rate etc. are hyper-parameters which basically means they are user specified, whereas ...
4
The Transformer model presented in this tutorial is an auto-regressive Transformer. Which means that prediction of next token only depends on it's previous tokens.
So in order to predict next token, you have to make sure that only previous token are attended. (If not, this would be a cheating because model already knows whats next).
So attention mask would ...
3
We give the target input into the transformer decoder while training the model. So it is easy for the model to "peek ahead" and learn what the next word would be. To ensure that this doesn't happen we apply an additive mask after the dot product between Query and Key. In the original paper "Attention is all you need", the triangular ...
3
Let's start with RNN. A well known problem is vanishin/exploding gradients, which means that the model is biased by most recent inputs in the sequence, or in other words, older inputs have practically no effect in the output at the current step.
LSTMs/GRUs mainly try to solve this problem, by including a separate memory (cell) and/or extra gates to learn ...
3
GPT-2 is a close copy of the basic transformer architecture.
GPT-2 does not require the encoder part of the original transformer architecture as it is decoder-only, and there are no encoder attention blocks, so the decoder is equivalent to the encoder, except for the MASKING in the multi-head attention block, the decoder is only allowed to glean information ...
2
Embeddings generated by transformers like Bert or XLM-R are fundamentally different from embeddings learned through language models like GloVe or Word2Vec.
The latter are static, i.e. they are just dictionaries containing a vocabulary with n-dimensional vectors associated to each word. Because of this they can be plotted through PCA and the distance between ...
2
I'm going to post another guess to this question - it won't be a complete answer, but hopefully it'll provide some direction towards finding a more legitimate answer.
The feed-forward networks as suggested by Vaswani are very reminiscent of the sparse autoencoders. Where the input / output dimensions are much greater than the hidden input dimension.
If you ...
2
1) The math is the exact same, so from an optimization or mathematical perspective there is no difference
2) Here are my guesses to a possible answer.
Habit: People may just call one over the other out of habit
Generality: Across frameworks a 1d convolution op would work, while Dense of FC may need adjustments to work on the temporal axis
Parallel ...
2
What is a transformer?
The original transformer, proposed in the paper Attention is all you need (2017), is an encoder-decoder-based neural network that is mainly characterized by the use of the so-called attention (i.e. a mechanism that determines the importance of words to other words in a sentence or which words are more likely to come together) and the ...
2
In statistics, if $X$ and $Y$ are independent and randomly distributed variables:
$\mathbb{E}[X + Y] = \mathbb{E}[X] + \mathbb{E}[Y] \\
Var(X + Y) = Var(X) + Var(Y) \\
\mathbb{E}[XY] = \mathbb{E}[X]\mathbb{E}[Y] \\
Var(XY) = (Var(X) + \mathbb{E}[X]^2)(Var(Y) + \mathbb{E}[Y]^2) - \mathbb{E}[X]^2\mathbb{E}[Y]^2$
Let $Q$ and $K$ be random $d_k$ x $d_k$ matrices,...
2
If you go through the main introductory paper of the transformer ("Attention is all you need"), you can find the comparison of the model with other state-of-the-art machine translation method:
For example, Deep-Att + PosUnk is a method that has utilized RNN and attention for the translation task. As you can see, the training cost for the ...
2
I have read the OpenNMT source code (https://github.com/OpenNMT/OpenNMT-py/blob/cd29c1dbfb35f4a2701ff52a1bf4e5bdcf02802e/onmt/modules/multi_headed_attn.py).
It seems like an extra linear layer learns the weights $W^{key}$ and $W^{value}$ (plus biases), so to get the output (keys and values), you multiply the output of the encoder's final add + norm layer by $...
2
I took a look at the Tensor2Tensor's source code implementation as per @nbro's suggestion, and it seems like the loss function is the cross entropy between the predicted ($\|sentence length\|$ x $\|vocab\|$) probability matrix (right before taking the argmax to find the token to output), and the $\|sentence length\|$-length vector of token IDs as the true ...
2
The reason each head is different is because they each learn a different set of weight matrices $\{ W_i^Q, W_i^K, W_i^V \}$ where $i$ is the index of the head. To clarify, the input to each attention head is the same. For attention head $i$:
$Q_i(x) = x W_i^Q \\
K_i(x) = x W_i^K \\
V_i(x) = x W_i^V \\
attention_i(x) = softmax \left(\frac{Q_i(x) K_i(x)^T}{\...
2
In short, repetition with feedback.
You are correct that machine learning (ML) models such as neural networks work with fixed dimensions for input and output. There are a few different ways to work around this when desired input and output is more variable. The most common approaches are:
Padding: Give the ML model capacity to cope with the largest expected ...
1
Yes, Transformers can be used to work with audio data, such as audio processing (audio classification, speaker identification, etc) (Audio ALBERT), speech-to-text (Streaming Automatic Speech Recognition with the Transformer Model), and text-to-speech (Neural Speech Synthesis with Transformer Network).
1
The masking should be applied to all Decoder blocks, otherwise in some blocks, past words can attend to future words, which would be cheating during training.
This is reflected in The Annotated Transformer as well. Notice that in the Decoder class, the forward function applies the same mask to each layer of the decoder:
class Decoder(nn.Module):
"...
1
I found the answer by reading the paper referenced by that section, Using the output embedding to improve language models
Based on this observation, we propose threeway weight tying (TWWT), where the input embedding of the decoder, the output embedding of
the decoder and the input embedding of the encoder are all tied. The single source/target vocabulary of ...
1
The authors of the original paper don't provide an explanation, but I suspect it's a combination of:
popular recognizable branding (cf. BERT, DALL-E, Watson etc)
similarity to [sequence] transduction / translation / transformations generally
1
If you're using a library such as Trax which contains great submodules for various Transformers (Skipping, BERT, Vanilla and Reformer) you can use the inbuilt trax.data.inputs.add_loss_weights() function and provide a value for the id_to_mask parameter.
Example Usage:
train_generator = trax.data.inputs.add_loss_weights(
data_generator(batch_size, x_train, ...
1
When it talks to other domains such as image or music, using transformer will always face a problem of sequence length limitation. To the best of my knowledge, the bottleneck of self-attention which uses a $n^2$ matrix quite limits transformer being applied to other domains. For example, a 32x32 pixel image, means a sequence of 1024 tokens.
OpenAI did some ...
1
This seems to be inherited from the original Google implementation, which also uses 2 outputs (https://github.com/google-research/bert/blob/master/run_pretraining.py#L293). I can see two possible reasons that the original implementation uses 2 outputs:
They are using the cross entropy loss, which typically works with log probabilities. To get probabilities ...
1
Are there examples that transformer have better accuracy than RNN end-to-end model like RNN-transducer for speech recognition?
Can transformer be used for online speech recognition which require low speech-end-to-result latency?
Does transformer have the potential to replace RNN end-to-end models for speech recognition in most cases in the future? This ...
1
Answer to Q1) If sampling for next token do you need to apply mask during inference?
Yes you do! The models ability to transfer information across positions was trained in this manner, and changing it up will have unpredictable consequences. Let my try to give an example:
Tokens: 1:sally, 2:sold, 3:seashells, 4:on, 5:the, 6:____
In the above you are ...
1
Can't see that this has been mentioned yet - there are ways to generate text non-sequentially using a non-autoregressive transformer, where you produce the entire response to the context at once. This typically produces worse accuracy scores because there are interdependencies within the text being produced - a model translating "thank you" could ...
Only top voted, non community-wiki answers of a minimum length are eligible
