Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [transformer]

For questions related to the transformer, which is a deep machine learning model introduced in 2017 in the paper "Attention Is All You Need", used primarily in the field of natural language processing (NLP).

Filter by
Sorted by
Tagged with
25 votes
1 answer
32k views

How does the (decoder-only) transformer architecture work?

How does the (decoder-only) transformer architecture work which is used in impressive models such as GPT-4?
Robin van Hoorn's user avatar
38 votes
2 answers
21k views

How can Transformers handle arbitrary length input?

The transformer, introduced in the paper Attention Is All You Need, is a popular new neural network architecture that is commonly viewed as an alternative to recurrent neural networks, like LSTMs and ...
chessprogrammer's user avatar
72 votes
4 answers
103k views

Why does the transformer do better than RNN and LSTM in long-range context dependencies?

I am reading the article How Transformers Work where the author writes Another problem with RNNs, and LSTMs, is that it’s hard to parallelize the work for processing sentences, since you have to ...
DRV's user avatar
  • 1,693
7 votes
1 answer
2k views

Why are biases (typically) not used in attention mechanism?

Watching this video implementing attention in a transformer. He set query, key, and value biases to False and said "Typically, people don't use biases for ...
Peyman's user avatar
  • 574
5 votes
2 answers
3k views

Is the Mask Needed for Masked Self-Attention During Inference with GPT-2

My understanding is that masked self-attention is necessary during training of GPT-2, as otherwise it would be able to directly see the correct next output at each iteration. My question is whether ...
D_s's user avatar
  • 51
25 votes
4 answers
9k views

Can the decoder in a transformer model be parallelized like the encoder?

Can the decoder in a transformer model be parallelized like the encoder? As far as I understand, the encoder has all the tokens in the sequence to compute the self-attention scores. But for a decoder,...
shiredude95's user avatar
20 votes
2 answers
13k views

Why does GPT-2 Exclude the Transformer Encoder?

After looking into transformers, BERT, and GPT-2, from what I understand, GPT-2 essentially uses only the decoder part of the original transformer architecture and uses masked self-attention that can ...
Athena Wisdom's user avatar
20 votes
3 answers
12k views

What kind of word embedding is used in the original transformer?

I am currently trying to understand transformers. To start, I read Attention Is All You Need and also this tutorial. What makes me wonder is the word embedding used in the model. Is word2vec or GloVe ...
Bert Gayus's user avatar
15 votes
3 answers
3k views

Why are embeddings added, not concatenated?

Let's consider the following example from BERT I cannot understand why "the input embeddings are the sum of the token embeddings, the segmentation embeddings, and the position embeddings". ...
nalzok's user avatar
  • 321
12 votes
2 answers
2k views

Why don't people use nonlinear activation functions after projecting the query key value in attention?

Why don't people use nonlinear activation functions after projecting the query key value in attention? It seems like doing this would lead to much-needed nonlinearity, otherwise, we're just doing ...
user3180's user avatar
  • 628
11 votes
3 answers
9k views

What is the purpose of Decoder mask (triangular mask) in Transformer?

I'm trying to implement transformer model using this tutorial. In the decoder block of the Transformer model, a mask is passed to "pad and mask future tokens in the input received by the decoder&...
Uchiha Madara's user avatar
8 votes
4 answers
6k views

Why does this multiplication of $Q$ and $K$ have a variance of $d_k$, in scaled dot product attention?

In scaled dot product attention, we scale our outputs by dividing the dot product by the square root of the dimensionality of the matrix: The reason why is stated that this constrains the ...
Jacob B's user avatar
  • 247
6 votes
1 answer
224 views

Why are "Transformers" called this way?

What is the reason behind the name "Transformers", for Multi Head Self-Attention-based neural networks from Attention is All You Need? I have been googling this question for a long time, and ...
Leevo's user avatar
  • 305
4 votes
1 answer
1k views

How do Transformer decoders handle arbitrary length input?

I am working through a Tensorflow Neural Machine Translation tutorial (https://www.tensorflow.org/text/tutorials/transformer) and am confused about how the decoder handles inputs when making ...
Dylan Larrabee's user avatar
3 votes
2 answers
5k views

Why do transformers have a fixed input length?

From what I understand, Transformer Encoders and Decoders use a fixed number of tokens as input, e.g., 512 tokens. In NLP for instance, different text sentences have a different number of tokens, and ...
A. Maman's user avatar
  • 131
2 votes
1 answer
557 views

Is the multi-head attention in the transformer a weighted adjacency matrix?

Are multi-head attention matrices weighted adjacency matrices? The job of the multi-head-attention mechanism in transformer models is to determine how likely a word is to appear after another word. In ...
MrPlanck's user avatar
  • 113
2 votes
0 answers
59 views

What part of the Vaswani et al. is the "transformer"?

Which part of this is the transformer? Ok, the caption says the whole thing is the transformer, but that's back in 2017 when the paper was published. My question is about how the community uses the ...
Alexander Soare's user avatar
2 votes
0 answers
265 views

How do the sine and cosine functions encode position in the transformer?

After going through both the "Illustrated Transformer" and "Annotated Transformer" blog posts, I still don't understand how the sinusoidal encodings are representing the position of elements in the ...
shoshi's user avatar
  • 121
2 votes
2 answers
2k views

What if we drop the causal mask in auto-regressive Transformer?

I understand the triangular causal mask in the attention is used to prevent tokens from "looking into the future", but why do we want to prevent that? Let's suppose we have a model with ...
nalzok's user avatar
  • 321
2 votes
2 answers
1k views

In the multi-head attention mechanism of the transformer, why do we need both $W_i^Q$ and ${W_i^K}^T$?

In the Attention is all you need paper, on the 4th page, we have equation 1, which describes the self-attention mechanism of the transformer architecture $$ \text { Attention }(Q, K, V)=\operatorname{...
Uğurcan Özalp's user avatar
1 vote
0 answers
82 views

How do transformer-based architectures generate contextual embeddings?

How do transformer-based architectures like Roberta generate contextual embeddings? The articles I've read keep saying that transformer encoders work bidirectionally. Because of self-attention, they ...
user avatar
1 vote
0 answers
126 views

Why do both sine and cosine have been used in positional encoding in the transformer model?

The Transformer model proposed in "Attention Is All You Need" uses sinusoid functions to do the positional encoding. Why have both sine and cosine been used? And why do we need to separate the odd ...
Shiyu's user avatar
  • 11
0 votes
1 answer
538 views

In the attention mechanism, why don't we normalize after multiplying values?

As this question says: In scaled dot product attention, we scale our outputs by dividing the dot product by the square root of the dimensionality of the matrix: The reason why is stated that this ...
Peyman's user avatar
  • 574
0 votes
1 answer
1k views

How to assess if OpenAI's ChatGPT chatbot has a human in the loop? [closed]

I've asked a question and given a couple answers that propose the OpenAI ChatGPT chatbot has humans in the loop (HITL), and that explains the chatbot's extraordinary abilities. I've been repeatedly ...
yters's user avatar
  • 387