8
I'll list some bullet points of the main innovations introduced by transformers , followed by bullet points of the main characteristics of the other architectures you mentioned, so we can then compared them.
Transformers
Transformes (Attention is all you need) were introduced in the context of machine translation with the purpose to avoid recursion in order ...
7
Let's start with a bit of notation and a couple of important clarification.
Q refers to the query vectors matrix, $q_i$ being a single query vector associated with a single input word.
V refers to the values vectors matrix, $v_i$ being a single value vector associated with a single input word.
K refers to the keys vectors matrix, $k_i$ being a single key ...
2
Embeddings generated by transformers like Bert or XLM-R are fundamentally different from embeddings learned through language models like GloVe or Word2Vec.
The latter are static, i.e. they are just dictionaries containing a vocabulary with n-dimensional vectors associated to each word. Because of this they can be plotted through PCA and the distance between ...
2
Can the decoder in a transformer model be parallelized like the
encoder?
NO:
Your understanding is completely right. In the decoder, the output of each step is fed to the bottom decoder in the next time step, just like an LSTM.
Also, like in LSTMs, the self-attention layer needs to attend to earlier positions in the output sequence in order to compute ...
2
Let's start with RNN. A well known problem is vanishin/exploding gradients, which means that the model is biased by most recent inputs in the sequence, or in other words, older inputs have practically no effect in the output at the current step.
LSTMs/GRUs mainly try to solve this problem, by including a separate memory (cell) and/or extra gates to learn ...
2
Parameters is a synonym for weights, which is the term most people use for a neural networks parameters (and indeed in my experience it is a term that machine learners will use in general whereas parameters is more often found in statistics literature). Batch size, learning rate etc. are hyper-parameters which basically means they are user specified, whereas ...
1
When it talks to other domains such as image or music, using transformer will always face a problem of sequence length limitation. To the best of my knowledge, the bottleneck of self-attention which uses a $n^2$ matrix quite limits transformer being applied to other domains. For example, a 32x32 pixel image, means a sequence of 1024 tokens.
OpenAI did some ...
1
This seems to be inherited from the original Google implementation, which also uses 2 outputs (https://github.com/google-research/bert/blob/master/run_pretraining.py#L293). I can see two possible reasons that the original implementation uses 2 outputs:
They are using the cross entropy loss, which typically works with log probabilities. To get probabilities ...
1
Are there examples that transformer have better accuracy than RNN end-to-end model like RNN-transducer for speech recognition?
Can transformer be used for online speech recognition which require low speech-end-to-result latency?
Does transformer have the potential to replace RNN end-to-end models for speech recognition in most cases in the future? This ...
1
Answer to Q1) If sampling for next token do you need to apply mask during inference?
Yes you do! The models ability to transfer information across positions was trained in this manner, and changing it up will have unpredictable consequences. Let my try to give an example:
Tokens: 1:sally, 2:sold, 3:seashells, 4:on, 5:the, 6:____
In the above you are ...
1
Basically, it means that the "localization network" should output a set of real valued parameters (typically 6 numbers). The word "regression" doesn't bear any specific meaning.
Any network that relies on the original image as input (directly or indirectly) and outputs 6 numbers, would work. And its last layer would qualify as "regression layer" as long as ...
1
I'm going to post another guess to this question - it won't be a complete answer, but hopefully it'll provide some direction towards finding a more legitimate answer.
The feed-forward networks as suggested by Vaswani are very reminiscent of the sparse autoencoders. Where the input / output dimensions are much greater than the hidden input dimension.
If you ...
1
1) The math is the exact same, so from an optimization or mathematical perspective there is no difference
2) Here are my guesses to a possible answer.
Habit: People may just call one over the other out of habit
Generality: Across frameworks a 1d convolution op would work, while Dense of FC may need adjustments to work on the temporal axis
Parallel ...
1
Can the decoder in a transformer model be parallelized like the encoder?
The correct answer is: computation in a Transformer decoder can be parallelized during training, but not during actual translation.
What exactly is parallelized?
Also, it's worth mentioning that "parallelization" in this case means to compute encoder or decoder states in paralllel ...
Only top voted, non community-wiki answers of a minimum length are eligible
