Questions tagged [transformer]
For questions related to the transformer, which is a deep machine learning model introduced in 2017 in the paper "Attention Is All You Need", used primarily in the field of natural language processing (NLP).
154
questions with no upvoted or accepted answers
8
votes
2
answers
843
views
Are there transformer-based architectures that can produce fixed-length vector encodings given arbitrary-length text documents?
BERT encodes a piece of text such that each token (usually words) in the input text map to a vector in the encoding of the text. However, this makes the length of the encoding vary as a function of ...
4
votes
1
answer
875
views
What are the keys and values of the attention model for the encoder and decoder in the "Attention Is All You Need" paper?
I have recently encountered the paper on NLP. It is very new to me and I am still unable to see how that works. I have used all the resources over there from the original paper to Youtube videos and ...
3
votes
1
answer
2k
views
Should I be layer freezing when fine-tuning an LLM?
I've had it in my head that generally speaking, it's better to freeze layers when fine-tuning an LLM, as per this quote from HuggingFace's article:
PEFT approaches only fine-tune a small number of (...
3
votes
0
answers
140
views
Why do LLMs like GPT-3 or Bloom use Vanilla Transformer instead of long sequence variants like Transformer-XL?
Is there any particular reason that the most recent and successful large language models like GPT-3 or Bloom utilize a vanilla Transformer architecture instead of an arguably superior long sequence ...
3
votes
0
answers
2k
views
What is input (and shape) to K/V/Q of self-attention of EACH Decoder block of Language-translation model Transformer's tokens during Inference?
Transformer model of the original Attention paper has a decoder unit that works differently during Inference than Tranining.
I'm trying to understand the shapes used during decoder (both self-...
3
votes
0
answers
429
views
Is there any point in adding the position embedding to the class token in Transformers?
The popular implementations of ViTs by Ross Wightman and Phil Wang add the position embedding to the class tokens as well as to the patches.
Is there any point in doing so?
The purpose of introduction ...
3
votes
0
answers
516
views
How to Select Model Parameters for Transformer (Heads, number of layers, etc)
Is there a general guideline on how the Transformer model parameters should be selected, or the range of these parameters that should be included in a hyperparameter sweep?
Number of heads
Number of ...
3
votes
0
answers
450
views
How to use TPU for real-time low-latency inference?
I use Google's Cloud TPU hardware extensively using Tensorflow for training models and inference, however, when I run inference I do it in large batches. The TPU takes about 3 minutes to warm up ...
2
votes
0
answers
76
views
How do transformer-based architectures generate contextual embeddings?
How do transformer-based architectures like Roberta generate contextual embeddings? The articles I've read keep saying that transformer encoders work bidirectionally. Because of self-attention, they ...
2
votes
0
answers
88
views
What is the meaning of "dimensionality of the embeddings and hidden states"?
I was reading the GPT-2 and LSTM documents and noticed that they use the terms "dimension of embedding and hidden state". For GPT-2, the size is $768$, and for LSTM, the size is $256$. What ...
2
votes
0
answers
76
views
Why is an encoder + decoder model with L by L layers the same speed as as decoder only model with 2 L layers?
I was watching this lecture: https://youtu.be/27rNqGrTdSI?t=2295
In it the presenter stated that:
"An encoder + decoder model with L by L layers is actually the same speed as as decoder only ...
2
votes
0
answers
262
views
Why is it said transformers are more parallelizable than RNN's?
The parallelization of transformers and RNNs (Recurrent Neural Networks) is often discussed. It's commonly said that transformers are more parallelizable than RNNs. However, this is a rather vague ...
2
votes
0
answers
61
views
Is it possible for original Vision Transformer (ViT) to do fine-grained semanantic segmentation? if so, how?
As far as I know, in the original ViT, the image is first divided to a fixed size of patch (16x16, for example) then they are flattened and treated as tokens and fed into Transformer.
Without using ...
2
votes
0
answers
340
views
What's the most efficient way of performing batched training of Causal Language Models?
I have seen a number of ways to train (yes, train, not fine-tune) these models efficiently with batches. I will illustrate these techniques with the following example dataset and context window:
...
2
votes
1
answer
1k
views
If GPT-3 is trained on predicting the next token, how is it able to take commands?
From my understanding, GPT-3 is trained on predicting the next token from a sequence of tokens. Given this, how is it able to take commands? For instance, in this example input, wouldn't the ...
2
votes
0
answers
52
views
Using similarity score within lstm embedding for attention based mechanism
Yesterday, I found this fascinating paper about predicting various clinical conditions using an attention based LSTM. I don't have any practical experience with attention mechanism or transformers, ...
2
votes
0
answers
352
views
How to include additional numeric input in the Transformer architecture
I want to apply the Transformer architecture to my machine translation task, and provide the decoder with an additional parameter in the range of [0,1]. This ...
2
votes
0
answers
556
views
Positional Encoding in Transformer on multi-variate time series data hurts performance
I set up a transformer model that embeds positional encodings in the encoder. The data is multi-variate time series-based data.
As I just experiment with the positional encoding portion of the code I ...
2
votes
0
answers
40
views
Are there any successful applications of transformers of small size (<10k weights)?
In the problems of NLP and sequence modeling, the Transformer architectures based on the self-attention mechanism (proposed in Attention Is All You Need) have achieved impressive results and now are ...
2
votes
0
answers
26
views
Embedding from Transformer-based model from paragraph or documnet (like Doc2Vec)
I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this:
...
2
votes
0
answers
57
views
What part of the Vaswani et al. is the "transformer"?
Which part of this is the transformer?
Ok, the caption says the whole thing is the transformer, but that's back in 2017 when the paper was published. My question is about how the community uses the ...
2
votes
0
answers
310
views
How to handle long sequences with transformers?
I have a time series sequence with 10 million steps. In step $t$, I have a 400 dimensional feature vector $X_t$ and a scalar value $y_t$ which I want to predict during inference time and I know during ...
2
votes
0
answers
40
views
Why does the loss stops reducing after a point in this Transformer Model?
Context
I was making a Transformer Model to convert English Sentences to German Sentences. But the loss stops reducing after some time.
Code
...
2
votes
0
answers
264
views
How are weight matrices in attention learned?
I have been looking into transformers lately and have been reading tons of tutorials. All of them address the intuition behind attention, which I understand. However, they treat learning the weight ...
2
votes
0
answers
181
views
What is the memory complexity of the memory-efficient attention in Reformer?
When I read the paper, Reformer: The Efficient Transformer, I cannot get the same complexity of the memory-efficient method in Table 1 (p. 5), which summarizes time/memory complexity of scaled dot-...
2
votes
0
answers
92
views
How to understand the matrices used in the Attention layer?
Attention-scoring mechanism seems to be a commonly-used component in various seq2seq models, and I was reading about the original "Location-based Attention" in Bahadanau well-known paper at https://...
2
votes
0
answers
174
views
Can you use transformer models to do autocomplete tasks?
I've researched online and seen many papers on the use of RNNs (like LSTMs or GRUs) to autocomplete for, say, a search engine, character by character. Which makes sense since it inherently predicts ...
2
votes
0
answers
1k
views
What is the time complexity of the forward pass and back-propagation of the sequence-to-sequence model with and without attention?
I keep looking through the literature, but can't seem to find any information regarding the time complexity of the forward pass and back-propagation of the sequence-to-sequence RNN encoder-decoder ...
2
votes
0
answers
163
views
How does positional encoding work in the transformer model?
In the transformer model, to incorporate positional information of texts, the researchers have added a positional encoding to the model. How does positional encoding work? How does the positional ...
2
votes
0
answers
339
views
Pretrained Models for Keyword-Based Text Generation
I'm looking for an implementation that allows me to generate text based on a pre-trained model (e.g. GPT-2).
An example would be gpt-2-keyword-generation (click here for demo). As the author notes, ...
2
votes
0
answers
640
views
Where should we place layer normalization in a transformer model?
In Attention Is All You Need paper:
That is, the output of each sub-layer is $LayerNorm(x+Sublayer(x))$, where $Sublayer(x)$ is the function implemented by the sub-layer itself. We apply dropout to ...
2
votes
0
answers
376
views
Why is the transformer for time series forecasting faster than RNN?
I've been reading different papers which implements the Transformer for time series forecasting. Most of the them are claiming that the training time is significantly faster then using a normal RNN. ...
2
votes
0
answers
29
views
How to train a transformer text-to-text model on counterexamples?
Is it possible to update the weights of a vanilla transformer model using counterexamples alongside examples?
For example, from the PAWS data set, given the phrases "Although interchangeable, the ...
2
votes
0
answers
374
views
How to interpret a large variance of the loss function?
How do I interpret a large variance of a loss function?
I am currently training a transformer network (using the software, but not the model from GPT-2) from scratch and my loss function looks like ...
2
votes
0
answers
264
views
How do the sine and cosine functions encode position in the transformer?
After going through both the "Illustrated Transformer" and "Annotated Transformer" blog posts, I still don't understand how the sinusoidal encodings are representing the position of elements in the ...
1
vote
0
answers
25
views
Is there any standardized notation for drawing neural network diagrams?
Is there any standardized notation for drawing neural network diagrams? For example, for circuits there is a universal set of symbols used to draw different types of circuits why not for neural ...
1
vote
1
answer
349
views
Is Softmax Necessary as the Activation Function for Self-Attention Mechanisms?
I’m curious about the mathematical reasoning behind the use of the softmax function as the activation function in self-attention mechanisms within neural networks. Specifically, I’m interested in ...
1
vote
0
answers
42
views
What is the input to an encoder-decoder transformer in next word prediction task?
I'm trying to understand how encoder-decoder architectures are used, or if they are used at all, for generative tasks that do not require an explicit prompt (ie. machine translation, summarization, ...
1
vote
0
answers
25
views
Modifying Cross Entropy Loss to work with multiple correct target sequences?
Let's say I'm training a transformer model to perform a seq to seq task, but there are multiple correct answers. For example, the following outputs would all be considered correct:
source: A B C -> ...
1
vote
1
answer
165
views
Masking in Decoder of Transformer
I understand that the masked multi-head attention block ensures that generation of token at time step t doesn't rely on subsequent tokens of the input. But the residual connection which adds the input ...
1
vote
0
answers
162
views
Concatenation of Feature vectors in transformers before passing to fcnn
** As I am new to the field , the question might feel little abstract and naïve considering my experience.
I am studying the Transformer architecture and trying to understand the various components ...
1
vote
1
answer
1k
views
What is considered the pre-fill, and what is considered the decoding phase in this process?
I've seen conflicting information about this online so I'm looking for clarification. I'm dealing with the causal LLaMAF model specifically.
I used to think that a sequence of tokens is generated in, ...
1
vote
1
answer
205
views
Is there any reference about backpropagation of the Transformer's multi-head layer?
Is there any reference about backpropagation of the Transformer's multi-head layer or multi-head attention (MHA)? I have searched various journals but have not found one yet.
1
vote
0
answers
567
views
What is MLM & NSP loss function
Two objective functions are used during the BERT language
model pretraining step.
The first one is masked language
model (MLM) that randomly masks
15% of the
input tokens and the objective is to ...
1
vote
0
answers
78
views
Why would increasing layers in PyTorch Transformer significantly increase loss?
I have a simple torch.nn.Transformer module for machine translation on the Multi30k dataset. It performs pretty well (32.2 Bleu score) but I looked at scaling up ...
1
vote
1
answer
64
views
Does Number of Fully connected neural networks changes in transformer architechture based on max length input size?
Considering the architecture of encoder and decoder in transformer as shown below:
Does each input token after self attention mechanism (z1,z2,z3,...)is passed to it's specific separate Feed forward ...
1
vote
0
answers
98
views
In terms of explainability, is attentive RNN easier to explain than the transformer?
Although the multi-headed attention block of the transformer allows the model to be more expressive (and therefore perform better), it is remarkably more difficult to decompose and therefore to ...
1
vote
0
answers
45
views
How can the Transformer model tell from positional encoding data to the origional data?
I am having trouble understanding positional encoding. Say after the wor2vec or some encoding algo we get the tensor $[0.7, 0.4, 0.2]$ for the second position. Now the final input into the model would ...
1
vote
0
answers
43
views
Can I reduce computation by only predicting response tokens in a transformer and still get the same gradients?
I have been looking at the source code of the Stanford Alpaca model and I believe that during inference, the whole instruction + response data is fed into the model normally. Then the instruction part ...
1
vote
0
answers
196
views
How code analysis part of ChatGPT works and trained
ChatGPT can explain given code snippet we also ask question like "What does this variable do" , "Why this is used" and all. I gave C++ function snippet from an popular Open Source ...