Questions tagged [transformer]
For questions related to the transformer, which is a deep machine learning model introduced in 2017 in the paper "Attention Is All You Need", used primarily in the field of natural language processing (NLP).
367
questions
72
votes
4
answers
100k
views
Why does the transformer do better than RNN and LSTM in long-range context dependencies?
I am reading the article How Transformers Work where the author writes
Another problem with RNNs, and LSTMs, is that it’s hard to parallelize the work for processing sentences, since you have to ...
36
votes
2
answers
20k
views
How can Transformers handle arbitrary length input?
The transformer, introduced in the paper Attention Is All You Need, is a popular new neural network architecture that is commonly viewed as an alternative to recurrent neural networks, like LSTMs and ...
27
votes
1
answer
16k
views
How is BERT different from the original transformer architecture?
As far as I can tell, BERT is a type of Transformer architecture. What I do not understand is:
How is Bert different from the original transformer architecture?
What tasks are better suited for BERT,...
27
votes
1
answer
21k
views
What exactly are the "parameters" in GPT-3's 175 billion parameters and how are they chosen/generated?
When I studied neural networks, parameters were learning rate, batch size etc. But even GPT3's ArXiv paper does not mention anything about what exactly the parameters are, but gives a small hint that ...
25
votes
4
answers
9k
views
Can the decoder in a transformer model be parallelized like the encoder?
Can the decoder in a transformer model be parallelized like the encoder?
As far as I understand, the encoder has all the tokens in the sequence to compute the self-attention scores. But for a decoder,...
23
votes
1
answer
29k
views
How does the (decoder-only) transformer architecture work?
How does the (decoder-only) transformer architecture work which is used in impressive models such as GPT-4?
22
votes
2
answers
11k
views
Why would you implement the position-wise feed-forward network of the transformer with convolution layers?
The Transformer model introduced in "Attention is all you need" by Vaswani et al. incorporates a so-called position-wise feed-forward network (FFN):
In addition to attention sub-layers, each of the ...
20
votes
1
answer
11k
views
What is the intuition behind the dot product attention?
I am watching the video Attention Is All You Need by Yannic Kilcher.
My question is: what is the intuition behind the dot product attention?
$$A(q,K, V) = \sum_i\frac{e^{q.k_i}}{\sum_j e^{q.k_j}} v_i$$...
20
votes
2
answers
13k
views
Why does GPT-2 Exclude the Transformer Encoder?
After looking into transformers, BERT, and GPT-2, from what I understand, GPT-2 essentially uses only the decoder part of the original transformer architecture and uses masked self-attention that can ...
19
votes
3
answers
12k
views
What kind of word embedding is used in the original transformer?
I am currently trying to understand transformers.
To start, I read Attention Is All You Need and also this tutorial.
What makes me wonder is the word embedding used in the model. Is word2vec or GloVe ...
14
votes
3
answers
3k
views
Why are embeddings added, not concatenated?
Let's consider the following example from BERT
I cannot understand why "the input embeddings are the sum of the token embeddings, the segmentation embeddings, and the position embeddings". ...
12
votes
1
answer
5k
views
In Computer Vision, what is the difference between a transformer and attention?
Having been studying computer vision for a while, I still cannot understand what the difference between a transformer and attention is?
12
votes
1
answer
6k
views
Why does a transformer not use an activation function following the multi-head attention layer?
I was hoping someone could explain to me why in the transformer model from the "Attention is all you need" paper there is no activation applied after both the multihead attention layer and ...
11
votes
3
answers
9k
views
What is the purpose of Decoder mask (triangular mask) in Transformer?
I'm trying to implement transformer model using this tutorial. In the decoder block of the Transformer model, a mask is passed to "pad and mask future tokens in the input received by the decoder&...
11
votes
1
answer
2k
views
Why don't people use nonlinear activation functions after projecting the query key value in attention?
Why don't people use nonlinear activation functions after projecting the query key value in attention?
It seems like doing this would lead to much-needed nonlinearity, otherwise, we're just doing ...
10
votes
2
answers
7k
views
What is different in each head of a multi-head attention mechanism?
I have a difficult time understanding the "multi-head" notion in the original transformer paper. What makes the learning in each head unique? Why doesn't the neural network learn the same ...
8
votes
4
answers
6k
views
Why does this multiplication of $Q$ and $K$ have a variance of $d_k$, in scaled dot product attention?
In scaled dot product attention, we scale our outputs by dividing the dot product by the square root of the dimensionality of the matrix:
The reason why is stated that this constrains the ...
8
votes
2
answers
2k
views
Can someone help me understand the intuition behind the query, key and value matrices in the transformer architecture?
I have been working mechanically with transformers, hoping that with time clarity about what the query, key, and value matrices represent will develop; but I am still lost. Would greatly benefit from ...
8
votes
2
answers
3k
views
What is the difference between the positional encoding techniques of the Transformer and GPT?
I know the original Transformer and the GPT (1-3) use two slightly different positional encoding techniques.
More specifically, in GPT they say positional encoding is learned. What does that mean? ...
8
votes
2
answers
4k
views
"Attention is all you need" paper : How are the Q, K, V values calculated?
The seminal Attention is all you need paper (Google Brain team, 2017) introduces Transformers and implements the attention mecanism with "queries, keys, values", in an analogy to a retrieval ...
8
votes
2
answers
5k
views
Why class embedding token is added to the Visual Transformer?
In the famous work on the Visual Transformers, the image is split into patches of a certain size (say 16x16), and these patches are treated as tokens in the NLP tasks. In order to perform ...
8
votes
2
answers
858
views
Are there transformer-based architectures that can produce fixed-length vector encodings given arbitrary-length text documents?
BERT encodes a piece of text such that each token (usually words) in the input text map to a vector in the encoding of the text. However, this makes the length of the encoding vary as a function of ...
7
votes
1
answer
4k
views
Is there a proper initialization technique for the weight matrices in multi-head attention?
Self-attention layers have 4 learnable tensors (in the vanilla formulation):
Query matrix $W_Q$
Key matrix $W_K$
Value matrix $W_V$
Output matrix $W_O$
Nice illustration from https://jalammar....
7
votes
1
answer
730
views
How do we know if GPT-2 is a better language model?
You may have heard of GPT2, a new language model. It has recently attracted attention from the general public as the foundation that published the paper, OpenAI, ironically refused to share the whole ...
6
votes
1
answer
2k
views
Why are biases (typically) not used in attention mechanism?
Watching this video implementing attention in a transformer. He set query, key, and value biases to False and said "Typically, people don't use biases for ...
6
votes
1
answer
5k
views
What is the cost function of a transformer?
The paper Attention Is All You Need describes the transformer architecture that has an encoder and a decoder.
However, I wasn't clear on what the cost function to minimize is for such an architecture.
...
6
votes
1
answer
4k
views
When exactly does the split into different heads in Multi-Head-Attention occur?
I am confused by the Multi-Head part of the Multi-Head-Attention used in Transformers. My question concerns the implementations in Pytorch of nn.MultiheadAttention and its forward method ...
6
votes
1
answer
215
views
Why are "Transformers" called this way?
What is the reason behind the name "Transformers", for Multi Head Self-Attention-based neural networks from Attention is All You Need?
I have been googling this question for a long time, and ...
5
votes
3
answers
1k
views
Can you confirm that the transformer works strictly deterministically and there is no randomness inside or between the attention layers?
On a high-level temperature and randomness affect the output of a generative language model:
Lower temperature: Produces more focused, conservative, and consistent responses.
Moderate temperature: ...
5
votes
3
answers
3k
views
How do open source LLMs compare to GPT-4?
I have heard some back and forth regarding open source LLMs like Llama.
I have heard that on certain benchmarks they perform close, the same or better than GPT-4, but caveats that they tend to lack ...
5
votes
2
answers
3k
views
Why do Transformers have a sequence limit at inference time?
As far as I understand, Transformer's time complexity increases quadratically with respect to the sequence length. As a result, during training to make training feasible, a maximum sequence limit is ...
5
votes
3
answers
2k
views
What is the Intermediate (dense) layer in between attention-output and encoder-output dense layers within transformer block in PyTorch implementation?
In PyTorch, transformer (BERT) models have an intermediate dense layer in between attention and output layers whereas the BERT and Transformer papers just mention the attention connected directly to ...
5
votes
2
answers
2k
views
Is the Mask Needed for Masked Self-Attention During Inference with GPT-2
My understanding is that masked self-attention is necessary during training of GPT-2, as otherwise it would be able to directly see the correct next output at each iteration. My question is whether ...
5
votes
2
answers
1k
views
What is the intuition behind self-attention?
I've been watching a few lectures on transformers, especially for language translation, though it seemingly becomes more confusing the more I watch.
In this lecture, there seems to be two conflicting ...
5
votes
2
answers
320
views
What makes a transformer a transformer?
Transformers are modified heavily in recent research. But what exactly makes a transformer a transformer? What is the core part of a transformer? Is it the self-attention, the parallelism, or ...
5
votes
2
answers
2k
views
What is the weight matrix in self-attention?
I've been looking into self-attention lately, and in the articles that I've been seeing, they all talk about "weights" in attention. My understanding is that the weights in self-attention ...
5
votes
2
answers
3k
views
How is the next token predicted in transformers?
In the transformer (or GPT/decoder only), at the end of the decoder blocks but before the final linear layer you have X vectors (for the X tokens at the input of the decoder). We then want to compute ...
5
votes
2
answers
938
views
Transformers: how does the decoder final layer output the desired token?
In the paper Attention Is All You Need, this section confuses me:
In our model, we share the same weight matrix between the two embedding layers [in the encoding section] and the pre-softmax linear ...
4
votes
2
answers
1k
views
Why different noise in GAN generate different images?
I understand that noise $z$ serves as the input to the generator. Noise $z$ is essentially a vector of random numbers, typically from Gaussian distribution with chosen size of like $100$. However, I ...
4
votes
1
answer
779
views
How can Transformers handle random sequences?
I have asked ChatGPT the following:
Can you concatenate jfef9230rj2mreg90r23ewfrn02eqwdk and
32ir20r3i2ofg90r32kee?
And without any error the model produces:
...
4
votes
1
answer
2k
views
Which situation will helpful using encoder or decoder or both in transformer model?
I have some questions about using (encoder / decoder / encoder-decoder) transformer models, included (language) transformer or Vision transformer.
The overall form of a transformer consists of an ...
4
votes
2
answers
6k
views
While fine-tuning a decoder only LLM like LLaMA on chat dataset, what kind of padding should one use?
While fine-tuning a decoder only LLM like LLaMA on chat dataset, what kind of padding should one use?
Many papers use Left Padding, but is right padding wrong since transformers gives the following ...
4
votes
1
answer
16k
views
Why can't AI image generators output verbatim text when prompted to do so?
I want to create a splash screen that includes the name of my project. DALL-E 2 changed some of the letters in the name, even when I tried putting the name of my project in double-quotes (...
4
votes
1
answer
2k
views
Do Vision Transformers handle arbitrary sequence lengths the same way as normal Transformers?
Does ViT do handle arbitrary sequence lengths using masking the same way the normal Transformer does?
The ViT paper doesn't mention anything about it, so I assume it uses masking like the normal ...
4
votes
3
answers
757
views
Recent deep learning textbook (i.e. covering at least GANs, LSTM and transformers and attention)
I am searching for an academic (i.e. with maths formulae) textbook which covers (at least) the following:
GAN
LSTM and transformers (e.g. seq2seq)
Attention mechanism
The closest match I got is Deep ...
4
votes
1
answer
2k
views
What is the proper way to process continuous sequence data, such as time-series, using the Transformer?
What is the right way to input continuous, temporal (time-series) data into the Transformer? Assume we're using the basic TransformerBlock here.
Since data is continuous with no tokens, Token ...
4
votes
2
answers
803
views
How does a transformer leverage the GPU to be trained faster than RNNs?
How does a transformer leverage the GPU to be trained faster than RNNs?
I understand the parameter space of the transformer might be significantly larger than that of the RNN. But why does the ...
4
votes
1
answer
562
views
Difference between dot product attention and "matrix attention"
As far as I know, attention was first introduced in Learning To Align And Translate.
There, the core mechanism which is able to disregard the sequence length, is a dynamically-built matrix, of shape ...
4
votes
1
answer
1k
views
How do Transformer decoders handle arbitrary length input?
I am working through a Tensorflow Neural Machine Translation tutorial (https://www.tensorflow.org/text/tutorials/transformer) and am confused about how the decoder handles inputs when making ...
4
votes
1
answer
1k
views
How does the embeddings work in vision transformer from paper?
I get the part from the paper where the image is split into P say 16x16 (smaller images) patches and then you have to ...