Questions tagged [transformer]
For questions related to the transformer, which is a deep machine learning model introduced in 2017 in the paper "Attention Is All You Need", used primarily in the field of natural language processing (NLP).
393
questions
0
votes
1
answer
11
views
How often do you have to recalculate the hidden states of each token while running inference?
We have a transformer with a single layer, a single head, an embedding dimension of 8 and a vocab size of 10. That makes W_k,W_v,W_q and W_ff all 8x8 and the out_ff 8x10. Now we run inference twice on ...
- 125
0
votes
0
answers
10
views
How is positional encoding done in wav2vec 2.0?
The "wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations" quickly describes the positional embedding method with:
Instead of fixed positional
embeddings which ...
- 101
0
votes
1
answer
28
views
Is the feed forward layer in the last encoder block of a transformer unnecessary?
I am trying to wrap my head around the transformer model using this great resource:
https://jalammar.github.io/illustrated-transformer/
I may need some clarification.
My understanding is that we take ...
- 103
1
vote
0
answers
17
views
Are there freely available pre-trained "toy models" of transformers suitable for inspecting residual stream?
I'm currently deeply invested in the Transformer Circuits thread in parallel with 3blue1brown's videos (chapter 7 on the MLP layer was released a day or two ago) to gain a better theoretical ...
- 111
0
votes
0
answers
22
views
What is the current state of the art in video transformers (mainly for tasks like classification) and what are the Top 5 papers from the last 2 years?
Is there a general consensus in the community regarding
the most effective video transformer architecture
which modalities to use, how to represent them, and the best methods for fusing them
the ...
1
vote
1
answer
32
views
What is the difference between transformer types?
These transformer types are "encoder only", "encoder-decoder", and "decoder only". What's the difference between them?
- 39
0
votes
1
answer
43
views
How does a transformer choose the next word concretely?
I am quite new to the topic of transformers. In a lecture of mine, we defined transformers as sequence to sequence mappings, i.e. a transformer $\mathcal{T}$ takes $X \in \mathbb{R}^{n \times d}$ and ...
- 103
0
votes
0
answers
21
views
In the Original Transformer, How exactly is layer normalization done?
I am assuming we are talking about the Transformer as written in "Attention is all you need" paper.
Let's say the in the normalization layer is m x n, where m is the context size and n is ...
- 243
1
vote
0
answers
24
views
Causal attention with left padding
I am trying to train a decoder-only transformer model. The dataset is left-padded to a fixed length so sequences of tokens can be batched. However, when I try to pass input through a multi head ...
- 11
0
votes
0
answers
17
views
In a vision transformer, are the patch outputs for the last layer unused?
In a vision transformer (https://arxiv.org/pdf/2010.11929 ), it seems like the final MLP head for prediction is attached only to the last layer's [cls] token embedding (Figure 1 and Eqn 4).
Does this ...
- 136
0
votes
1
answer
27
views
How to convert a positionally encoded predicted embedding from a decoder to its matching token?
Is it valid to just subtract the positional encoding from a predicted output if the decoder was also positionally encoded? Or does masking take care of this problem, and the decoder should only learn ...
0
votes
0
answers
15
views
Sequence to Sequence vs Sequence to Token
I came here to ask for some clarification about the subject that is in the topic.
Denote: Seq2Seq, Seq2Tok
What I am trying to understand if there is any use of the output sequence in Seq2Seq models ...
- 101
0
votes
0
answers
31
views
How can I change the tokens BERT uses so that each digit is a separate token?
Rather than have the tokenizer generate this sort of thing:
"$1009 Dollars" => ["$", "100#", "9", "Dollars"]
I'...
- 101
0
votes
0
answers
23
views
Summary Generation
I want to create a summary from a list of some bullet points and keywords . Most NLP and Transformer based models are not very well suited for short sentences and bullet point.
Bullet points are ...
- 1
0
votes
0
answers
22
views
Transformer Loss Function for Music Generation
I am working on a Midi Generation project that takes tracks as inputs, and outputs a complimentary track of notes.
The tracks are basically a list of notes created of:
Time
Duration
Pitch
Velocity
I ...
- 1
2
votes
2
answers
173
views
Are there any non-transformer LLMs?
Almost all LLMs are based on the transformer architecture, but are there any examples of ones that don't use transformers?
user84165
0
votes
1
answer
30
views
Combinig output of two different machine learning models for accurate invoice data extraction: Is this a viable approach?
I am working (trying to work) on a project to extract relevant information from invoices. Currently I don't achieve much good accuracy so am trying to come up with some new ideas. I am considering ...
- 1
0
votes
0
answers
28
views
How to add/embed categorical features in transformers network?
I would like to give more context to my transformers by adding some metadata related to each token. This metadata is mostly categorical (3 fields, with 3 possible values for each field).
In addition ...
- 1
0
votes
0
answers
25
views
Normalizing the embedding space of an encoder language model with respect to categorical data
Suppose we have a tree/hierarchy of categories (e.g. categories of products in an e-commerce website), each node being assigned a title. Assume that the title of each node is semantically accurate, ...
- 101
0
votes
2
answers
75
views
How do Transformer models ensure unique token representations when combining embeddings and positional encodings?
In Transformer models, token embeddings are combined with positional encodings through element-wise addition to incorporate positional information. However, this raises a concern about the potential ...
- 603
0
votes
0
answers
44
views
How does casual and padding mask work in decoder-only models?
I am trying to implement a decoder-only model from scratch using PyTorch, but I am confused about how the masking works. From what I understand, when we have encoder-decoder architecture, the padding ...
1
vote
1
answer
52
views
Would converting embeddings updated by a transformer into tokens (eg: by searching for the nearest embeddings) produce results that make sense?
In the transformer architecture, one of the step is to update the embeddings, allowing words to pass information to whichever words they are relevant to. For example in the sentence "a small cat ...
- 111
1
vote
1
answer
97
views
Filter scientific papers based on abstract information
I have a data frame that contains different numbers of columns, the important ones are:
Title Abstract
The length (number of words) varies between each row, with a ...
- 113
0
votes
1
answer
65
views
How do transformer models handle negation in sentiment analysis
I'm trying to understand how transformer models, such as BERT or GPT, handle negation in sentiment analysis. Specifically, I'm curious about how these models manage to correctly interpret sentences ...
- 101
0
votes
1
answer
51
views
How is the bidirectional context achieved in BERT?
I have read the paper "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding" by Jacob Devlin et al. (2018) and "Improving Language Understanding by ...
- 3
0
votes
0
answers
45
views
Is a small transformer model able to effectively handle any input length provided it is fine-tuned on it?
Suppose we have a transformer LLM which can do a task such as summarising.
I know transformer can technically handle any input length (assume we are not using learned positional embeddings) because ...
- 1
1
vote
0
answers
20
views
What about the loss and custom metric with per-pair weights in multi-class classification?
Let's suppose that we have a multi-class classification problem with 5 classes: 0, 1, 2, 3, 4. The order is not random, they are neighbors. For example, imagine that a labelling is 1. If the ...
- 111
0
votes
2
answers
54
views
Classifier-Free-Guidance with Transformers
I'm working on music generation using transformers.
Using the decoder part for the audio tokens with text conditioning by the T5 encoder
In Classifier-Free-Guidance, the text conditioning randomly ...
- 1
0
votes
0
answers
78
views
Why decoder only model require left padding?
We used Gemma 2B model to infer, and tried left and right padding. "right" padding is giving us different answer compared to left padding.
Why do we use left padding for decoder only model, ...
- 45
0
votes
0
answers
31
views
Why I am getting different KV cache?
I have taken Squad 2.0 dataset for inferencing Gemma 2B model.
When I provided model with 1st datapoint truncating till 36 tokens and same datapoint truncating till 80 tokens.
I am getting slightly ...
- 45
2
votes
1
answer
76
views
Will an AI LLM learn a language if fed during training with a large corpus of undeciphered language?
AIs can learn many languages just by being trained on their corpora.
What will happen if we in addition would train it on a large corpus of undeciphered language, like Minoan or Etruscan? Will it be ...
- 351
1
vote
2
answers
89
views
Can transformer attention make predictions based on analogy?
Suppose I have included 3 examples of an idiosyncratic sentence for training by a transformer:
Example 1: Asdfogiug likes Zsdfoiusdhf and Zsdfoiusdhf likes Asdfogiug too.
Example 2: Bsodifhas likes ...
- 157
0
votes
0
answers
25
views
Unexpected results using ORPO trl
For studying purposes, I've created a very small dataset about a fictional city called "Auryn":
https://huggingface.co/datasets/celsowm/auryn_dpo_orpo_english
So, my goal is to "inject&...
- 101
0
votes
1
answer
22
views
Spatial vs spatiotemporal methods for object counting in low frame-rate videos
I'm currently working on an object counting/density estimation task using low frame rate video (~2 fps) in a traffic setting. I've explored a lot of literature on both spatial methods (i.e. using only ...
- 11
0
votes
1
answer
63
views
Why are the Q and K matrices two separate matrices in attention?
If I understand correctly the attention layer is represented as
$$
\begin{align}
&softmax(\frac{Q K^T}{\sqrt{d_k}}) V \\
= &softmax(\frac{(s W_q) (s W_k)^T}{\sqrt{d_k}}) V \\
= &softmax(\...
- 103
1
vote
0
answers
37
views
What’s more efficient in multihead attention: multiply QKV by $W_i$ then split or linearly project QKV $h$ times into dimensions $d_k$?
I’m looking to bridge two implementations of multihead attention.
Approach 1: Multiply and Split
Each of the queries, keys, and values is multiplied by a separate square weight matrix of size (...
- 11
0
votes
0
answers
35
views
Why can't we use only keys to calculate self-attention?
I was reading about the self-attention mechanism and the paper suggests to have 3 things to be computed: Key, Query and Value. As far as I understood the reason for having Value is to allow ...
0
votes
0
answers
24
views
How does BERT know how to where to add segment embeddings (i.e. to differentiate between two sentences packed in a single token sequence)
In addition to using a special [SEP] token to distinguish between two sentences, I understand that BERT also adds special learned embeddings to each sentence:
"we add a learned embedding to every ...
- 1
0
votes
0
answers
11
views
Incredibly High CrossEntropyLoss in Sequence-to-Sequence Generation
I'm trying to do SMILES chemical representation prediction from a large dataset (Around 5M Samples) to teach it do predict another downstream task. The model's part responsible for generating the data ...
- 101
0
votes
1
answer
50
views
Determining optimal data size for generalization in transformer encoders, particularly for Time-Series signal data
I'm currently experimenting with training a model that employs a single transformer encoder on time-series signal data. Despite having a relatively small dataset of around 50 examples, each with a ...
- 409
2
votes
1
answer
156
views
What should be Relationship between embedding dimension and context length?
What should we keep hidden dimension/embedding dimension (d_model as per attention is all you need paper), greater, equal, or smaller to the context length (n)?
Is there any such relationship between ...
- 45
0
votes
1
answer
108
views
xLSTM parallel computation - mismatch in dimensions
In this recent paper, a new architecture is proposed, called xLSTM. I've implemented the sequential version in PyTorch, but it's slower than I would like, so I'm now implementing the parallel version ...
- 101
0
votes
1
answer
20
views
Understanding different methods of covariance parametrization
In the paper https://arxiv.org/pdf/2112.02143 it is noted that there are different ways to "parametrize" covariance. (page 4)
What does "parametrizing" covariance mean exactly? In ...
0
votes
1
answer
100
views
What does "position-wise" fully connected mean?
I understand the architecture of a position-wise feed-forward network as described in (https://nn.labml.ai/transformers/feed_forward.html). However I do not understand what "position wise" ...
0
votes
0
answers
18
views
When using local self-attention how do keys and queries wind up having compatible dimensions if if different sized convolutions are used to make them?
In the paper https://arxiv.org/pdf/2112.02143 on page 3 in figure 1 an architecture is described.
As can be seen in the upper left, the keys for Local self-attention are generated using a 3x3 ...
1
vote
1
answer
137
views
Is there a relationship between tokens and parameters in LLMs?
What the question says.
In a transformer architecture, is there a relationship between number of tokens and number of parameters?
Can you have a LLM with a small number of parameters but a large ...
- 143
0
votes
0
answers
22
views
Is there any purpose of altering neural network architecture if validation loss does not decrease but training loss does?
I am training a transformer based neural network and the validation loss is not decreasing, but the training loss does decrease. I am wondering if it's possible to debug or change the architecture ...
- 223
0
votes
2
answers
49
views
What if the sum of word embedding and positional embedding becomes same for different words?
In Transformers, we add the positional embedding with the word embedding and then process it. But, what if the sum of the embeddings become same for different words at different position of a sentence?...
1
vote
0
answers
76
views
How to Interpret Cross Attention
I am a bit confused on what cross attention mechanisms are doing. I understand that the currently decoded output is usually the query and the conditioning/input (from an encoder) is the key and value. ...
0
votes
0
answers
37
views
why explicit reference of batchsize is required in attention
When I read the code for multi-head attention, I noticed all people consider the batch_size in the forward()
For example, the code below was provided by GPT:
...
- 1