Questions tagged [transformer]

For questions related to the transformer, which is a deep machine learning model introduced in 2017 in the paper "Attention Is All You Need", used primarily in the field of natural language processing (NLP).

Filter by
Sorted by
Tagged with
0 votes
0 answers
4 views

Can you illustrate how the weights in transformer model generated from a training sentence can be generalized to an unseen test sentence?

Can you show how the weights in transformer model are generalizable?
Steven's user avatar
  • 101
0 votes
0 answers
10 views

Masking during Instruction Tuning for LLM finetuning

I'm currently trying to learn more about LLMs particularly generative decoder only models such as the GPT family of models. I do have one question about masking though. For me the way masking is ...
Pile_of_linear_Algebra's user avatar
0 votes
1 answer
32 views

When are Transformers better than LSTMs in time-series tasks such as classification?

I’m working on a time-series classification problem and trying to decide whether to use a Transformer or an LSTM. From what I’ve learned, Transformers are better suited for capturing long-range ...
Mark Cortejo's user avatar
0 votes
0 answers
22 views

Prefix tuning in LLM uses learnable vectors to fine tune the model

I would like to implement a new architecture for Transformer. Below description is my thought. Prefix tuning in LLM uses learnable vectors to fine tune the model. Is there a way to use the output ...
jackson's user avatar
0 votes
1 answer
41 views

Understanding the transformer at inference time

Let's consider language translation and let $I_1,\ldots,I_{N_i}$ be the $N_i$ input tokens. My understanding is that the encoder produces $N_i$ embeddings, which I will refer to as $E_1,\ldots,E_{N_i}$...
Felix Crazzolara's user avatar
1 vote
0 answers
20 views

What is the input to an encoder-decoder transformer in next word prediction task?

I'm trying to understand how encoder-decoder architectures are used, or if they are used at all, for generative tasks that do not require an explicit prompt (ie. machine translation, summarization, ...
mehsheenman's user avatar
1 vote
0 answers
14 views

Modifying Cross Entropy Loss to work with multiple correct target sequences?

Let's say I'm training a transformer model to perform a seq to seq task, but there are multiple correct answers. For example, the following outputs would all be considered correct: source: A B C -> ...
Brayden Alexander Rudisill's user avatar
0 votes
0 answers
32 views

Why is masked self attention necessary on GPT decoder

I'm currently reading the paper for the first GPT model and I'm confused about why masked self attention is necessary and I haven’t found any good answers online. The consensus seems to be that we don'...
Kiran Manicka's user avatar
0 votes
1 answer
71 views

Why is it called multi-headed attention?

Why do we call the attention layer in transformers multi-headed attention when in practice all the attention matrices from different heads (W,K,V) for a single layer are concatenated to perform the ...
Tarique's user avatar
  • 103
1 vote
1 answer
88 views

Masking in Decoder of Transformer

I understand that the masked multi-head attention block ensures that generation of token at time step t doesn't rely on subsequent tokens of the input. But the residual connection which adds the input ...
SAGALPREET SINGH's user avatar
0 votes
0 answers
10 views

Are the outputs of layers in Attention Is All You Need interpretable by mapping to tokens?

In the basic transformer model from 2017, I'm a bit confused what the outputs of each layer are supposed to be. Are they embeddings? If so, does that mean you could examine a given output from a given ...
Stan Shunpike's user avatar
0 votes
0 answers
69 views

Why is a linear and not a non-linear transformation used in self-attention to calculate queries, keys and values?

In self-attention, one vector undergoes three different transformations to create the query, keys and values. These are always a simple linear transformations. Why is it considered sufficient? Shouldn'...
Kasia's user avatar
  • 193
0 votes
2 answers
70 views

Why encoders are required in Transformers

In the original Transformers paper why encoder is added when a decoder alone can do what an encoder can do (like multi-head attention, feed-forward NN etc....). I mean even a decoder also has the same ...
Swastik's user avatar
0 votes
2 answers
202 views

From where do the Encoders in Transformers gets Input Embedding from?

In Transformers Encoders, from where do the Encoders get Input Embedding from? So when a sentence is given to a transformer-based model it first tokenises the sentence and each token is mapped with ...
Swastik's user avatar
0 votes
1 answer
67 views

Time Series Classification using Transformer Encoder

Lets say I have a collection of tensors, each tensor representing a time series with 64 points and 4 features. The dimension of each tensor would be [64,4]. I am trying to classify these series. For ...
Zohaib Hamdule's user avatar
2 votes
1 answer
121 views

Aren't context lengths for transformers an artificial restriction?

Let's focus on the case of decoder-only transformers, where I am using algorithm 10 from "Formal Algorithms for Transformers" by Mary Phung and Marcus Hutter as a reference. : https://i....
Robert Wegner's user avatar
0 votes
1 answer
70 views

How to force Transformer to give more weight to certain tokens

I'm developing an encoder-decoder based transformer model and I would like to ask if there are ways to incentivize or penalize certain tokens during training. I'm working on a translation task where ...
jasperagrante's user avatar
1 vote
1 answer
78 views

Why is there a shared matrix W in graph attention networks instead of the query-key-value trio like in regular transformers?

In section 2.1 of the Graph attention network paper The graph attention layer is described as as an initial step, a shared linear transformation, parametrized by a weight matrix, W ∈ RF ′×F , is ...
oliver.c's user avatar
-1 votes
1 answer
236 views

How to get Llama-2 Rotary Embeddings?

I want to get the Llama-2 rotary embeddings. I do print(model) and get the following output: In the picture I highlight the rotary embeddings. How can get the ...
Christian01's user avatar
0 votes
0 answers
14 views

What is the policy model in RLHF for LLMs?

What is the policy model doing explicitly in an LLM with RLHF setup? From my understanding, LLMs generate in a way that is no different from any of their predecessors: beam search decoding, ...
information_interchange's user avatar
0 votes
1 answer
187 views

Transformer decoder. Causal masking during inference?

I understand how causal masking in the self-attention layer of the decoder works and why we use it during training. What I want to ask is: should we use causal masking during inference ? Consider a ...
pi-tau's user avatar
  • 612
1 vote
2 answers
116 views

Why is dot-product and not Euclidean distance used for attention?

In models using attention (eg Transformer architectures) we used scaled dot-product to measure similarity rather than (negative or inverse) Euclidean distance. Why is this the case? Does Layer ...
Betterthan Kwora's user avatar
2 votes
0 answers
61 views

Why is an encoder + decoder model with L by L layers the same speed as as decoder only model with 2 L layers?

I was watching this lecture: https://youtu.be/27rNqGrTdSI?t=2295 In it the presenter stated that: "An encoder + decoder model with L by L layers is actually the same speed as as decoder only ...
shawn's user avatar
  • 21
0 votes
1 answer
28 views

Do I need to manually segment/augment my data for transformer training?

If my dataset consists of the sentence "I have an apple" do I need to feed my model the separate examples "I" -> "have", "I have" -> "an", and &...
User's user avatar
  • 175
0 votes
0 answers
85 views

Does Embeddings and Vector Databases solve the need of having longer context windows?

I am learning to use the OpenAI API to build LLM-based agents. I recently came across the concept of vector databases, which use embeddings to convert text into vectors and store them in a database ...
Cesar Ruiz's user avatar
0 votes
0 answers
38 views

Backpropagation in a transformer

I have a transformer for timeseries forcasting based on this article https://arxiv.org/abs/2001.08317 Given a source containing $src=(x_{t-5},x_{t-4},x_{t-3},x_{t-2},x_{t-1})$ and a target of $tgt=(x_{...
Michał Kuczynski's user avatar
0 votes
1 answer
54 views

How can BERT/Transformer models accept input batches of different sizes?

I understand that all inputs in a batch need to be of the same size. However, it seems BERT/Transformers models can accept batches with different sizes as input. How is that possible? I thought we ...
PS1's user avatar
  • 101
0 votes
1 answer
41 views

How does GPT like Decoder only conversational models distunguish the source of text?

In a conversational setting where two sources of text (user and the model) follow each other like below User: some text bla bla Model: another text bah bah User: bla bla bla Model: bah bah and so on, ...
meliksahturker's user avatar
1 vote
0 answers
75 views

Concatenation of Feature vectors in transformers before passing to fcnn

** As I am new to the field , the question might feel little abstract and naïve considering my experience. I am studying the Transformer architecture and trying to understand the various components ...
Buddha Dev Bhattacharjee's user avatar
0 votes
0 answers
52 views

Can pretraining be continued after RLHF?

Assume you have a pretrained transformer language model (M1) which already underwent reinforcement learning by human feedback (M2). I assume that it is in principle possible to continue the ...
Hans-Peter Stricker's user avatar
0 votes
0 answers
20 views

What if in DPR (dense passage retrieval), the answer belongs to more than one passage?

In the DPR paper the dataset is expected to be in this format D = {<qi, pi+, pi,1-, ... >} With only one positive passage, but it is possible that the question requires an answer that spans ...
naren's user avatar
  • 1
0 votes
0 answers
22 views

Training Issue in Solving Multi-Dimensional Multiple Knapsack Problem with Transformer Model and PPO and SAC algorithm

I'm reaching out to the brilliant minds of the AI community to seek help with a challenging issue in my project on solving the multi-dimensional multiple knapsack problem using a transformer model. As ...
Mohammad Hosseini's user avatar
0 votes
1 answer
125 views

Why use exponential and log in Positional Encoding of Transformer

This code snippet is from here under the section named "Position embeddings". ...
Jun's user avatar
  • 3
0 votes
1 answer
126 views

Understanding embedding outputs in transformer models like CLIP

I'm working with OpenAI's CLIP model and trying to understand the output of the text encoder. When I input a short prompt like "cat", the output is a tensor of shape [77, 1024]. My ...
hanugm's user avatar
  • 3,612
0 votes
0 answers
80 views

Feature Importance for FT Transformer

I hope someone is able to shed light on this. I was reading through the codes on this link with regards to getting feature importance from attention scores. https://github.com/aruberts/...
DDM's user avatar
  • 101
1 vote
1 answer
34 views

diagonal of the hypercube in cross-attention

In Cross-attention, divide the similarity between $Q$ and $K$ by $\sqrt{d_k}$. Here $\sqrt{d_k}$ is the diagonal of the hypercube, is this a coincidence or famous theorem? $$ \frac{QK^T}{\sqrt{d_k}} $...
diffusion stable's user avatar
0 votes
1 answer
81 views

Confusion About Triangle Mask in Transformer Decoder

I have some confusion about the implementation of the triangle mask in the transformer decoder. I understand the reasoning for the mask, it prevents the network from 'cheating' by looking ahead at the ...
new2java's user avatar
1 vote
1 answer
704 views

Which situation will helpful using encoder or decoder or both in transformer model?

I have some questions about using (encoder / decoder / encoder-decoder) transformer models, included (language) transformer or Vision transformer. The overall form of a transformer consists of an ...
Yang's user avatar
  • 27
2 votes
2 answers
2k views

While fine-tuning a decoder only LLM like LLaMA on chat dataset, what kind of padding should one use?

While fine-tuning a decoder only LLM like LLaMA on chat dataset, what kind of padding should one use? Many papers use Left Padding, but is right padding wrong since transformers gives the following ...
basujindal's user avatar
1 vote
1 answer
155 views

Why in Multi-Head Attention implementation should we use $3$ linear layers for Q, K, V instead of $3 * h$ layers?

I have been trying to implement a Transformer architecture using PyTorch by following the Attention Is All You Need paper as well as the The Annotated Transformer blog post to compare my code with ...
Daviiid's user avatar
  • 573
1 vote
1 answer
125 views

Can transformers autoregressively generate a sequence of embeddings (instead of predictions)?

Is it theoretically possible to use a transformer architecture to autoregressively generate a sequence of embedding vectors, instead of discrete tokens? For example, if I were to provide an input of a ...
Theo Coombes's user avatar
1 vote
1 answer
72 views

Modern graduate-level machine learning books with focus on generative models

I'm looking for a modern machine learning book with graduate-level treatment of more recent topics such as diffusion and generative models, transformers etc. I have a hard copy of Deep Learning by ...
user74376's user avatar
  • 111
7 votes
2 answers
890 views

Can someone help me understand the intuition behind the query, key and value matrices in the transformer architecture?

I have been working mechanically with transformers, hoping that with time clarity about what the query, key, and value matrices represent will develop; but I am still lost. Would greatly benefit from ...
Chinmay's user avatar
  • 501
0 votes
0 answers
35 views

In DDPM model, why authors subtract the largest value along j dimension of q*k values matrices?

I have a question about a part of DDPM model code. Below is the Attention class of DDPM model from this website. My questions is at the bottom. ...
Jun's user avatar
  • 3
0 votes
0 answers
47 views

What are the *non-cost-related* reasons RNN+Attention underperform Transformers?

There are obvious trainability and performance challenges with RNNs, such as having to process in serial and BPTT. But let's say we magically had an "optimal" set of weights for the RNN + ...
llllvvuu's user avatar
  • 146
1 vote
2 answers
138 views

How is the padding mask incorporated in the attention formula?

I have been looking for the answer in other questions but no one tackled that. I want to ask you how is the padding mask considered in the formula of attention? The attention formula taking into ...
Daviiid's user avatar
  • 573
0 votes
1 answer
586 views

Why does LLM inference cost scale in both input tokens and output tokens?

EDIT This question was flawed. See my answer with help from commenters. Original question This question has been asked in other forums [1] [2] but I'm not sure I understand the claims, which are (...
llllvvuu's user avatar
  • 146
0 votes
0 answers
104 views

Does anyone recognize this formula to quantify the likelihood that a transformer will generate the same response twice?

The idea is simple enough. Just multiply the likelihood of filling in the blank with the same result as the original response. $$\prod_{s:substring}^{t:string}P(t|masked(t,s))$$ Motivation: Rather ...
Andrew Johnson's user avatar
1 vote
0 answers
36 views

Why does averaging attention-weighted positions reduce the effective resolution in transformers?

I was reading this blog post from Harvard and it says in its background paragraph about transformers that the number of operations required to relate signals from two arbitrary input or output ...
Daviiid's user avatar
  • 573
1 vote
1 answer
691 views

What is considered the pre-fill, and what is considered the decoding phase in this process?

I've seen conflicting information about this online so I'm looking for clarification. I'm dealing with the causal LLaMAF model specifically. I used to think that a sequence of tokens is generated in, ...
jgeddes's user avatar
  • 11

1
2 3 4 5
7