Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [transformer]

For questions related to the transformer, which is a deep machine learning model introduced in 2017 in the paper "Attention Is All You Need", used primarily in the field of natural language processing (NLP).

165 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
8 votes
2 answers
876 views

Are there transformer-based architectures that can produce fixed-length vector encodings given arbitrary-length text documents?

BERT encodes a piece of text such that each token (usually words) in the input text map to a vector in the encoding of the text. However, this makes the length of the encoding vary as a function of ...
HelloGoodbye's user avatar
4 votes
1 answer
945 views

What are the keys and values of the attention model for the encoder and decoder in the "Attention Is All You Need" paper?

I have recently encountered the paper on NLP. It is very new to me and I am still unable to see how that works. I have used all the resources over there from the original paper to Youtube videos and ...
Deshwal's user avatar
  • 263
3 votes
1 answer
3k views

Should I be layer freezing when fine-tuning an LLM?

I've had it in my head that generally speaking, it's better to freeze layers when fine-tuning an LLM, as per this quote from HuggingFace's article: PEFT approaches only fine-tune a small number of (...
multiheadedattention's user avatar
3 votes
0 answers
150 views

Why do LLMs like GPT-3 or Bloom use Vanilla Transformer instead of long sequence variants like Transformer-XL?

Is there any particular reason that the most recent and successful large language models like GPT-3 or Bloom utilize a vanilla Transformer architecture instead of an arguably superior long sequence ...
hokage555's user avatar
3 votes
0 answers
2k views

What is input (and shape) to K/V/Q of self-attention of EACH Decoder block of Language-translation model Transformer's tokens during Inference?

Transformer model of the original Attention paper has a decoder unit that works differently during Inference than Tranining. I'm trying to understand the shapes used during decoder (both self-...
Joe Black's user avatar
  • 181
3 votes
0 answers
466 views

Is there any point in adding the position embedding to the class token in Transformers?

The popular implementations of ViTs by Ross Wightman and Phil Wang add the position embedding to the class tokens as well as to the patches. Is there any point in doing so? The purpose of introduction ...
spiridon_the_sun_rotator's user avatar
3 votes
0 answers
539 views

How to Select Model Parameters for Transformer (Heads, number of layers, etc)

Is there a general guideline on how the Transformer model parameters should be selected, or the range of these parameters that should be included in a hyperparameter sweep? Number of heads Number of ...
Athena Wisdom's user avatar
3 votes
0 answers
460 views

How to use TPU for real-time low-latency inference?

I use Google's Cloud TPU hardware extensively using Tensorflow for training models and inference, however, when I run inference I do it in large batches. The TPU takes about 3 minutes to warm up ...
adng's user avatar
  • 51
2 votes
0 answers
82 views

Why is an encoder + decoder model with L by L layers the same speed as as decoder only model with 2 L layers?

I was watching this lecture: https://youtu.be/27rNqGrTdSI?t=2295 In it the presenter stated that: "An encoder + decoder model with L by L layers is actually the same speed as as decoder only ...
shawn's user avatar
  • 21
2 votes
0 answers
298 views

Why is it said transformers are more parallelizable than RNN's?

The parallelization of transformers and RNNs (Recurrent Neural Networks) is often discussed. It's commonly said that transformers are more parallelizable than RNNs. However, this is a rather vague ...
Erick Macias's user avatar
2 votes
0 answers
61 views

Is it possible for original Vision Transformer (ViT) to do fine-grained semanantic segmentation? if so, how?

As far as I know, in the original ViT, the image is first divided to a fixed size of patch (16x16, for example) then they are flattened and treated as tokens and fed into Transformer. Without using ...
wanburana's user avatar
2 votes
0 answers
388 views

What's the most efficient way of performing batched training of Causal Language Models?

I have seen a number of ways to train (yes, train, not fine-tune) these models efficiently with batches. I will illustrate these techniques with the following example dataset and context window: ...
thesofakillers's user avatar
2 votes
0 answers
200 views

How code analysis part of ChatGPT works and trained

ChatGPT can explain given code snippet we also ask question like "What does this variable do" , "Why this is used" and all. I gave C++ function snippet from an popular Open Source ...
BRUCE's user avatar
  • 121
2 votes
1 answer
1k views

If GPT-3 is trained on predicting the next token, how is it able to take commands?

From my understanding, GPT-3 is trained on predicting the next token from a sequence of tokens. Given this, how is it able to take commands? For instance, in this example input, wouldn't the ...
Andrew Tang's user avatar
2 votes
0 answers
55 views

Using similarity score within lstm embedding for attention based mechanism

Yesterday, I found this fascinating paper about predicting various clinical conditions using an attention based LSTM. I don't have any practical experience with attention mechanism or transformers, ...
mAI's user avatar
  • 21
2 votes
0 answers
407 views

How to include additional numeric input in the Transformer architecture

I want to apply the Transformer architecture to my machine translation task, and provide the decoder with an additional parameter in the range of [0,1]. This ...
Felix Schön's user avatar
2 votes
0 answers
598 views

Positional Encoding in Transformer on multi-variate time series data hurts performance

I set up a transformer model that embeds positional encodings in the encoder. The data is multi-variate time series-based data. As I just experiment with the positional encoding portion of the code I ...
Matt's user avatar
  • 121
2 votes
0 answers
40 views

Are there any successful applications of transformers of small size (<10k weights)?

In the problems of NLP and sequence modeling, the Transformer architectures based on the self-attention mechanism (proposed in Attention Is All You Need) have achieved impressive results and now are ...
spiridon_the_sun_rotator's user avatar
2 votes
0 answers
26 views

Embedding from Transformer-based model from paragraph or documnet (like Doc2Vec)

I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this: ...
Bloodstone Programmer's user avatar
2 votes
0 answers
59 views

What part of the Vaswani et al. is the "transformer"?

Which part of this is the transformer? Ok, the caption says the whole thing is the transformer, but that's back in 2017 when the paper was published. My question is about how the community uses the ...
Alexander Soare's user avatar
2 votes
0 answers
326 views

How to handle long sequences with transformers?

I have a time series sequence with 10 million steps. In step $t$, I have a 400 dimensional feature vector $X_t$ and a scalar value $y_t$ which I want to predict during inference time and I know during ...
mhsnk's user avatar
  • 233
2 votes
0 answers
40 views

Why does the loss stops reducing after a point in this Transformer Model?

Context I was making a Transformer Model to convert English Sentences to German Sentences. But the loss stops reducing after some time. Code ...
NITIN AGARWAL's user avatar
2 votes
0 answers
271 views

How are weight matrices in attention learned?

I have been looking into transformers lately and have been reading tons of tutorials. All of them address the intuition behind attention, which I understand. However, they treat learning the weight ...
Ege Demir's user avatar
2 votes
0 answers
185 views

What is the memory complexity of the memory-efficient attention in Reformer?

When I read the paper, Reformer: The Efficient Transformer, I cannot get the same complexity of the memory-efficient method in Table 1 (p. 5), which summarizes time/memory complexity of scaled dot-...
tsu's user avatar
  • 471
2 votes
0 answers
98 views

How to understand the matrices used in the Attention layer?

Attention-scoring mechanism seems to be a commonly-used component in various seq2seq models, and I was reading about the original "Location-based Attention" in Bahadanau well-known paper at https://...
Joe Black's user avatar
  • 181
2 votes
0 answers
190 views

Can you use transformer models to do autocomplete tasks?

I've researched online and seen many papers on the use of RNNs (like LSTMs or GRUs) to autocomplete for, say, a search engine, character by character. Which makes sense since it inherently predicts ...
user1234544's user avatar
2 votes
0 answers
1k views

What is the time complexity of the forward pass and back-propagation of the sequence-to-sequence model with and without attention?

I keep looking through the literature, but can't seem to find any information regarding the time complexity of the forward pass and back-propagation of the sequence-to-sequence RNN encoder-decoder ...
user1234544's user avatar
2 votes
0 answers
165 views

How does positional encoding work in the transformer model?

In the transformer model, to incorporate positional information of texts, the researchers have added a positional encoding to the model. How does positional encoding work? How does the positional ...
Eka's user avatar
  • 1,066
2 votes
0 answers
339 views

Pretrained Models for Keyword-Based Text Generation

I'm looking for an implementation that allows me to generate text based on a pre-trained model (e.g. GPT-2). An example would be gpt-2-keyword-generation (click here for demo). As the author notes, ...
Comfort Eagle's user avatar
2 votes
0 answers
656 views

Where should we place layer normalization in a transformer model?

In Attention Is All You Need paper: That is, the output of each sub-layer is $LayerNorm(x+Sublayer(x))$, where $Sublayer(x)$ is the function implemented by the sub-layer itself. We apply dropout to ...
Alexey Romanov's user avatar
2 votes
0 answers
391 views

Why is the transformer for time series forecasting faster than RNN?

I've been reading different papers which implements the Transformer for time series forecasting. Most of the them are claiming that the training time is significantly faster then using a normal RNN. ...
razvanc92's user avatar
  • 1,148
2 votes
0 answers
29 views

How to train a transformer text-to-text model on counterexamples?

Is it possible to update the weights of a vanilla transformer model using counterexamples alongside examples? For example, from the PAWS data set, given the phrases "Although interchangeable, the ...
adng's user avatar
  • 51
2 votes
0 answers
390 views

How to interpret a large variance of the loss function?

How do I interpret a large variance of a loss function? I am currently training a transformer network (using the software, but not the model from GPT-2) from scratch and my loss function looks like ...
allo's user avatar
  • 310
2 votes
0 answers
265 views

How do the sine and cosine functions encode position in the transformer?

After going through both the "Illustrated Transformer" and "Annotated Transformer" blog posts, I still don't understand how the sinusoidal encodings are representing the position of elements in the ...
shoshi's user avatar
  • 121
1 vote
0 answers
22 views

How to Interpret Cross Attention

I am a bit confused on what cross attention mechanisms are doing. I understand that the currently decoded output is usually the query and the conditioning/input (from an encoder) is the key and value. ...
Kiran Manicka's user avatar
1 vote
0 answers
12 views

Decision Transformer: more than a "trajectory picking" algorithm?

I'm studying the Decision Transformer for some offline reinforcement learning tasks. The basic idea is to collect a huge quantity of data generated by a real experimental device (let's say an arm ...
Dave's user avatar
  • 155
1 vote
0 answers
17 views

Compare two songs content using Audio Spectogram Transformer

I'm trying to establish a similarity metric between two songs. To do this I'm using the AST model on HuggingFace. This model basically works in a way very similar to a ViT but applied to spectograms ...
user491880's user avatar
1 vote
0 answers
37 views

Is there any standardized notation for drawing neural network diagrams?

Is there any standardized notation for drawing neural network diagrams? For example, for circuits there is a universal set of symbols used to draw different types of circuits why not for neural ...
play's user avatar
  • 33
1 vote
0 answers
82 views

How do transformer-based architectures generate contextual embeddings?

How do transformer-based architectures like Roberta generate contextual embeddings? The articles I've read keep saying that transformer encoders work bidirectionally. Because of self-attention, they ...
user avatar
1 vote
1 answer
665 views

Is Softmax Necessary as the Activation Function for Self-Attention Mechanisms?

I’m curious about the mathematical reasoning behind the use of the softmax function as the activation function in self-attention mechanisms within neural networks. Specifically, I’m interested in ...
Kasia's user avatar
  • 303
1 vote
0 answers
55 views

What is the input to an encoder-decoder transformer in next word prediction task?

I'm trying to understand how encoder-decoder architectures are used, or if they are used at all, for generative tasks that do not require an explicit prompt (ie. machine translation, summarization, ...
thevoid's user avatar
  • 111
1 vote
0 answers
29 views

Modifying Cross Entropy Loss to work with multiple correct target sequences?

Let's say I'm training a transformer model to perform a seq to seq task, but there are multiple correct answers. For example, the following outputs would all be considered correct: source: A B C -> ...
Brayden Alexander Rudisill's user avatar
1 vote
1 answer
223 views

Masking in Decoder of Transformer

I understand that the masked multi-head attention block ensures that generation of token at time step t doesn't rely on subsequent tokens of the input. But the residual connection which adds the input ...
SAGALPREET SINGH's user avatar
1 vote
0 answers
228 views

Concatenation of Feature vectors in transformers before passing to fcnn

** As I am new to the field , the question might feel little abstract and naïve considering my experience. I am studying the Transformer architecture and trying to understand the various components ...
Buddha Dev Bhattacharjee's user avatar
1 vote
1 answer
2k views

What is considered the pre-fill, and what is considered the decoding phase in this process?

I've seen conflicting information about this online so I'm looking for clarification. I'm dealing with the causal LLaMAF model specifically. I used to think that a sequence of tokens is generated in, ...
jgeddes's user avatar
  • 11
1 vote
1 answer
263 views

Is there any reference about backpropagation of the Transformer's multi-head layer?

Is there any reference about backpropagation of the Transformer's multi-head layer or multi-head attention (MHA)? I have searched various journals but have not found one yet.
poglhar's user avatar
  • 23
1 vote
0 answers
692 views

What is MLM & NSP loss function

Two objective functions are used during the BERT language model pretraining step. The first one is masked language model (MLM) that randomly masks 15% of the input tokens and the objective is to ...
XYZ's user avatar
  • 121
1 vote
0 answers
87 views

Why would increasing layers in PyTorch Transformer significantly increase loss?

I have a simple torch.nn.Transformer module for machine translation on the Multi30k dataset. It performs pretty well (32.2 Bleu score) but I looked at scaling up ...
Matt Harrison's user avatar
1 vote
1 answer
68 views

Does Number of Fully connected neural networks changes in transformer architechture based on max length input size?

Considering the architecture of encoder and decoder in transformer as shown below: Does each input token after self attention mechanism (z1,z2,z3,...)is passed to it's specific separate Feed forward ...
Arjun Reddy's user avatar
1 vote
0 answers
101 views

In terms of explainability, is attentive RNN easier to explain than the transformer?

Although the multi-headed attention block of the transformer allows the model to be more expressive (and therefore perform better), it is remarkably more difficult to decompose and therefore to ...
hH1sG0n3's user avatar
  • 221