Questions tagged [transformer]
For questions related to the transformer, which is a deep machine learning model introduced in 2017 in the paper "Attention Is All You Need", used primarily in the field of natural language processing (NLP).
326
questions
0
votes
0
answers
9
views
Prefix tuning in LLM uses learnable vectors to fine tune the model
I would like to implement a new architecture for Transformer.
Below description is my thought.
Prefix tuning in LLM uses learnable vectors to fine tune the model.
Is there a way to use the output ...
- 1
0
votes
1
answer
35
views
Understanding the transformer at inference time
Let's consider language translation and let $I_1,\ldots,I_{N_i}$ be the $N_i$ input tokens. My understanding is that the encoder produces $N_i$ embeddings, which I will refer to as $E_1,\ldots,E_{N_i}$...
- 109
1
vote
0
answers
19
views
What is the input to an encoder-decoder transformer in next word prediction task?
I'm trying to understand how encoder-decoder architectures are used, or if they are used at all, for generative tasks that do not require an explicit prompt (ie. machine translation, summarization, ...
- 111
0
votes
0
answers
12
views
Modifying Cross Entropy Loss to work with multiple correct target sequences?
Let's say I'm training a transformer model to perform a seq to seq task, but there are multiple correct answers. For example, the following outputs would all be considered correct:
source: A B C -> ...
0
votes
0
answers
32
views
Why is masked self attention necessary on GPT decoder
I'm currently reading the paper for the first GPT model and I'm confused about why masked self attention is necessary and I haven’t found any good answers online.
The consensus seems to be that we don'...
0
votes
1
answer
70
views
Why is it called multi-headed attention?
Why do we call the attention layer in transformers multi-headed attention when in practice all the attention matrices from different heads (W,K,V) for a single layer are concatenated to perform the ...
- 103
1
vote
1
answer
87
views
Masking in Decoder of Transformer
I understand that the masked multi-head attention block ensures that generation of token at time step t doesn't rely on subsequent tokens of the input. But the residual connection which adds the input ...
- 107
0
votes
0
answers
9
views
Are the outputs of layers in Attention Is All You Need interpretable by mapping to tokens?
In the basic transformer model from 2017, I'm a bit confused what the outputs of each layer are supposed to be. Are they embeddings? If so, does that mean you could examine a given output from a given ...
- 111
0
votes
0
answers
66
views
Why is a linear and not a non-linear transformation used in self-attention to calculate queries, keys and values?
In self-attention, one vector undergoes three different transformations to create the query, keys and values. These are always a simple linear transformations. Why is it considered sufficient? Shouldn'...
- 193
0
votes
2
answers
69
views
Why encoders are required in Transformers
In the original Transformers paper why encoder is added when a decoder alone can do what an encoder can do (like multi-head attention, feed-forward NN etc....).
I mean even a decoder also has the same ...
- 1
0
votes
2
answers
193
views
From where do the Encoders in Transformers gets Input Embedding from?
In Transformers Encoders, from where do the Encoders get Input Embedding from?
So when a sentence is given to a transformer-based model it first tokenises the sentence and each token is mapped with ...
- 1
0
votes
1
answer
65
views
Time Series Classification using Transformer Encoder
Lets say I have a collection of tensors, each tensor representing a time series with 64 points and 4 features. The dimension of each tensor would be [64,4]. I am trying to classify these series. For ...
2
votes
1
answer
111
views
Aren't context lengths for transformers an artificial restriction?
Let's focus on the case of decoder-only transformers, where I am using algorithm 10 from "Formal Algorithms for Transformers" by Mary Phung and Marcus Hutter as a reference.
: https://i....
0
votes
1
answer
69
views
How to force Transformer to give more weight to certain tokens
I'm developing an encoder-decoder based transformer model and I would like to ask if there are ways to incentivize or penalize certain tokens during training.
I'm working on a translation task where ...
- 101
1
vote
1
answer
77
views
Why is there a shared matrix W in graph attention networks instead of the query-key-value trio like in regular transformers?
In section 2.1 of the Graph attention network paper
The graph attention layer is described as
as an initial step, a shared
linear transformation, parametrized by a weight matrix, W ∈ RF ′×F , is ...
- 35
-1
votes
1
answer
216
views
How to get Llama-2 Rotary Embeddings?
I want to get the Llama-2 rotary embeddings. I do print(model) and get the following output:
In the picture I highlight the rotary embeddings.
How can get the ...
- 99
0
votes
0
answers
14
views
What is the policy model in RLHF for LLMs?
What is the policy model doing explicitly in an LLM with RLHF setup?
From my understanding, LLMs generate in a way that is no different from any of their predecessors: beam search decoding, ...
0
votes
1
answer
170
views
Transformer decoder. Causal masking during inference?
I understand how causal masking in the self-attention layer of the decoder works and why we use it during training. What I want to ask is: should we use causal masking during inference ?
Consider a ...
- 602
1
vote
2
answers
111
views
Why is dot-product and not Euclidean distance used for attention?
In models using attention (eg Transformer architectures) we used scaled dot-product to measure similarity rather than (negative or inverse) Euclidean distance. Why is this the case?
Does Layer ...
- 121
2
votes
0
answers
61
views
Why is an encoder + decoder model with L by L layers the same speed as as decoder only model with 2 L layers?
I was watching this lecture: https://youtu.be/27rNqGrTdSI?t=2295
In it the presenter stated that:
"An encoder + decoder model with L by L layers is actually the same speed as as decoder only ...
- 21
0
votes
1
answer
28
views
Do I need to manually segment/augment my data for transformer training?
If my dataset consists of the sentence "I have an apple" do I need to feed my model the separate examples "I" -> "have", "I have" -> "an", and &...
- 175
0
votes
0
answers
83
views
Does Embeddings and Vector Databases solve the need of having longer context windows?
I am learning to use the OpenAI API to build LLM-based agents. I recently came across the concept of vector databases, which use embeddings to convert text into vectors and store them in a database ...
- 453
0
votes
0
answers
38
views
Backpropagation in a transformer
I have a transformer for timeseries forcasting based on this article https://arxiv.org/abs/2001.08317
Given a source containing $src=(x_{t-5},x_{t-4},x_{t-3},x_{t-2},x_{t-1})$ and a target of $tgt=(x_{...
- 101
0
votes
1
answer
46
views
How can BERT/Transformer models accept input batches of different sizes?
I understand that all inputs in a batch need to be of the same size. However, it seems BERT/Transformers models can accept batches with different sizes as input.
How is that possible? I thought we ...
- 101
0
votes
1
answer
41
views
How does GPT like Decoder only conversational models distunguish the source of text?
In a conversational setting where two sources of text (user and the model) follow each other like below
User: some text bla bla
Model: another text bah bah
User: bla bla bla
Model: bah bah
and so on, ...
- 296
1
vote
0
answers
69
views
Concatenation of Feature vectors in transformers before passing to fcnn
** As I am new to the field , the question might feel little abstract and naïve considering my experience.
I am studying the Transformer architecture and trying to understand the various components ...
0
votes
0
answers
47
views
Can pretraining be continued after RLHF?
Assume you have a pretrained transformer language model (M1) which already underwent reinforcement learning by human feedback (M2). I assume that it is in principle possible to continue the ...
0
votes
0
answers
18
views
What if in DPR (dense passage retrieval), the answer belongs to more than one passage?
In the DPR paper
the dataset is expected to be in this format D = {<qi, pi+, pi,1-, ... >}
With only one positive passage, but it is possible that the question requires an answer that spans ...
- 1
0
votes
0
answers
22
views
Training Issue in Solving Multi-Dimensional Multiple Knapsack Problem with Transformer Model and PPO and SAC algorithm
I'm reaching out to the brilliant minds of the AI community to seek help with a challenging issue in my project on solving the multi-dimensional multiple knapsack problem using a transformer model. As ...
0
votes
1
answer
118
views
Why use exponential and log in Positional Encoding of Transformer
This code snippet is from here under the section named "Position embeddings".
...
- 3
0
votes
1
answer
114
views
Understanding embedding outputs in transformer models like CLIP
I'm working with OpenAI's CLIP model and trying to understand the output of the text encoder. When I input a short prompt like "cat", the output is a tensor of shape [77, 1024]. My ...
- 3,612
0
votes
0
answers
77
views
Feature Importance for FT Transformer
I hope someone is able to shed light on this. I was reading through the codes on this link with regards to getting feature importance from attention scores.
https://github.com/aruberts/...
- 101
1
vote
1
answer
32
views
diagonal of the hypercube in cross-attention
In Cross-attention, divide the similarity between $Q$ and $K$ by $\sqrt{d_k}$.
Here $\sqrt{d_k}$ is the diagonal of the hypercube, is this a coincidence or famous theorem?
$$ \frac{QK^T}{\sqrt{d_k}} $...
0
votes
1
answer
76
views
Confusion About Triangle Mask in Transformer Decoder
I have some confusion about the implementation of the triangle mask in the transformer decoder. I understand the reasoning for the mask, it prevents the network from 'cheating' by looking ahead at the ...
- 1
1
vote
1
answer
658
views
Which situation will helpful using encoder or decoder or both in transformer model?
I have some questions about using (encoder / decoder / encoder-decoder) transformer models, included (language) transformer or Vision transformer.
The overall form of a transformer consists of an ...
- 27
2
votes
2
answers
2k
views
While fine-tuning a decoder only LLM like LLaMA on chat dataset, what kind of padding should one use?
While fine-tuning a decoder only LLM like LLaMA on chat dataset, what kind of padding should one use?
Many papers use Left Padding, but is right padding wrong since transformers gives the following ...
- 33
1
vote
1
answer
141
views
Why in Multi-Head Attention implementation should we use $3$ linear layers for Q, K, V instead of $3 * h$ layers?
I have been trying to implement a Transformer architecture using PyTorch by following the Attention Is All You Need paper as well as the The Annotated Transformer blog post to compare my code with ...
- 573
1
vote
1
answer
118
views
Can transformers autoregressively generate a sequence of embeddings (instead of predictions)?
Is it theoretically possible to use a transformer architecture to autoregressively generate a sequence of embedding vectors, instead of discrete tokens?
For example, if I were to provide an input of a ...
- 11
1
vote
1
answer
68
views
Modern graduate-level machine learning books with focus on generative models
I'm looking for a modern machine learning book with graduate-level treatment of more recent topics such as diffusion and generative models, transformers etc.
I have a hard copy of Deep Learning by ...
- 111
7
votes
2
answers
879
views
Can someone help me understand the intuition behind the query, key and value matrices in the transformer architecture?
I have been working mechanically with transformers, hoping that with time clarity about what the query, key, and value matrices represent will develop; but I am still lost. Would greatly benefit from ...
- 501
0
votes
0
answers
35
views
In DDPM model, why authors subtract the largest value along j dimension of q*k values matrices?
I have a question about a part of DDPM model code.
Below is the Attention class of DDPM model from this website. My questions is at the bottom.
...
- 3
0
votes
0
answers
47
views
What are the *non-cost-related* reasons RNN+Attention underperform Transformers?
There are obvious trainability and performance challenges with RNNs, such as having to process in serial and BPTT. But let's say we magically had an "optimal" set of weights for the RNN + ...
- 146
1
vote
2
answers
127
views
How is the padding mask incorporated in the attention formula?
I have been looking for the answer in other questions but no one tackled that. I want to ask you how is the padding mask considered in the formula of attention?
The attention formula taking into ...
- 573
0
votes
1
answer
561
views
Why does LLM inference cost scale in both input tokens and output tokens?
EDIT
This question was flawed. See my answer with help from commenters.
Original question
This question has been asked in other forums [1] [2] but I'm not sure I understand the claims, which are (...
- 146
0
votes
0
answers
103
views
Does anyone recognize this formula to quantify the likelihood that a transformer will generate the same response twice?
The idea is simple enough. Just multiply the likelihood of filling in the blank with the same result as the original response.
$$\prod_{s:substring}^{t:string}P(t|masked(t,s))$$
Motivation:
Rather ...
- 101
1
vote
0
answers
34
views
Why does averaging attention-weighted positions reduce the effective resolution in transformers?
I was reading this blog post from Harvard and it says in its background paragraph about transformers that the number of operations required to relate signals from two arbitrary input or output ...
- 573
1
vote
1
answer
656
views
What is considered the pre-fill, and what is considered the decoding phase in this process?
I've seen conflicting information about this online so I'm looking for clarification. I'm dealing with the causal LLaMAF model specifically.
I used to think that a sequence of tokens is generated in, ...
- 11
0
votes
0
answers
81
views
Computing confidence score for transformer output
I'm sampling multiple generations from a transformer model and I would like to have a confidence score for each generation. The Hugging Face library's generate() ...
- 141
2
votes
3
answers
2k
views
How do open source LLMs compare to GPT-4?
I have heard some back and forth regarding open source LLMs like Llama.
I have heard that on certain benchmarks they perform close, the same or better than GPT-4, but caveats that they tend to lack ...
- 133
0
votes
0
answers
11
views
Update OpenAI embedding based on own domain corpus
I have a large domain corpus. Is there anyway to update the word/sentence/document embedding obtained from OpenAI embedding API based on my own domain corpus? There may be some words in my corpus that ...
- 101