Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [transformer]

For questions related to the transformer, which is a deep machine learning model introduced in 2017 in the paper "Attention Is All You Need", used primarily in the field of natural language processing (NLP).

Filter by
Sorted by
Tagged with
0 votes
0 answers
4 views

Spatial vs spatiotemporal methods for object counting in low frame-rate videos

I'm currently working on an object counting/density estimation task using low frame rate video (~2 fps) in a traffic setting. I've explored a lot of literature on both spatial methods (i.e. using only ...
yuki's user avatar
  • 1
0 votes
1 answer
22 views

Why are the Q and K matrices two separate matrices in attention?

If I understand correctly the attention layer is represented as $$ \begin{align} &softmax(\frac{Q K^T}{\sqrt{d_k}}) V \\ = &softmax(\frac{(s W_q) (s W_k)^T}{\sqrt{d_k}}) V \\ = &softmax(\...
fakedrake's user avatar
  • 103
0 votes
0 answers
9 views

What’s more efficient in multihead attention: multiply QKV by W_i then split or linearly project QKV h times into dimensions d_k?

I’m looking to bridge two implementations of multihead attention. Approach 1: Multiply and Split Each of the queries, keys, and values is multiplied by a separate square weight matrix of size (...
marcocamilo's user avatar
0 votes
0 answers
14 views

Why can't we use only keys to calculate self-attention?

I was reading about the self-attention mechanism and the paper suggests to have 3 things to be computed: Key, Query and Value. As far as I understood the reason for having Value is to allow ...
Erik Nouroyan's user avatar
0 votes
0 answers
7 views

How does BERT know how to where to add segment embeddings (i.e. to differentiate between two sentences packed in a single token sequence)

In addition to using a special [SEP] token to distinguish between two sentences, I understand that BERT also adds special learned embeddings to each sentence: "we add a learned embedding to every ...
shan's user avatar
  • 1
0 votes
0 answers
9 views

Incredibly High CrossEntropyLoss in Sequence-to-Sequence Generation

I'm trying to do SMILES chemical representation prediction from a large dataset (Around 5M Samples) to teach it do predict another downstream task. The model's part responsible for generating the data ...
Vivek Joshy's user avatar
0 votes
1 answer
18 views

Determining optimal data size for generalization in transformer encoders, particularly for Time-Series signal data

I'm currently experimenting with training a model that employs a single transformer encoder on time-series signal data. Despite having a relatively small dataset of around 50 examples, each with a ...
Kulin Patel's user avatar
1 vote
1 answer
38 views

What should be Relationship between embedding dimension and context length?

What should we keep hidden dimension/embedding dimension (d_model as per attention is all you need paper), greater, equal, or smaller to the context length (n)? Is there any such relationship between ...
Aamod Thakur's user avatar
0 votes
1 answer
51 views

xLSTM parallel computation - mismatch in dimensions

In this recent paper, a new architecture is proposed, called xLSTM. I've implemented the sequential version in PyTorch, but it's slower than I would like, so I'm now implementing the parallel version ...
Quaere Verum's user avatar
0 votes
1 answer
15 views

Understanding different methods of covariance parametrization

In the paper https://arxiv.org/pdf/2112.02143 it is noted that there are different ways to "parametrize" covariance. (page 4) What does "parametrizing" covariance mean exactly? In ...
Homer Sanchez's user avatar
0 votes
1 answer
31 views

What does "position-wise" fully connected mean?

I understand the architecture of a position-wise feed-forward network as described in (https://nn.labml.ai/transformers/feed_forward.html). However I do not understand what "position wise" ...
Homer Sanchez's user avatar
0 votes
0 answers
15 views

When using local self-attention how do keys and queries wind up having compatible dimensions if if different sized convolutions are used to make them?

In the paper https://arxiv.org/pdf/2112.02143 on page 3 in figure 1 an architecture is described. As can be seen in the upper left, the keys for Local self-attention are generated using a 3x3 ...
Homer Sanchez's user avatar
1 vote
1 answer
69 views

Is there a relationship between tokens and parameters in LLMs?

What the question says. In a transformer architecture, is there a relationship between number of tokens and number of parameters? Can you have a LLM with a small number of parameters but a large ...
A. Darwin's user avatar
  • 133
0 votes
0 answers
19 views

Is there any purpose of altering neural network architecture if validation loss does not decrease but training loss does?

I am training a transformer based neural network and the validation loss is not decreasing, but the training loss does decrease. I am wondering if it's possible to debug or change the architecture ...
JobHunter69's user avatar
0 votes
2 answers
33 views

What if the sum of word embedding and positional embedding becomes same for different words?

In Transformers, we add the positional embedding with the word embedding and then process it. But, what if the sum of the embeddings become same for different words at different position of a sentence?...
Aakash Roy's user avatar
1 vote
0 answers
22 views

How to Interpret Cross Attention

I am a bit confused on what cross attention mechanisms are doing. I understand that the currently decoded output is usually the query and the conditioning/input (from an encoder) is the key and value. ...
Kiran Manicka's user avatar
0 votes
0 answers
29 views

why explicit reference of batchsize is required in attention

When I read the code for multi-head attention, I noticed all people consider the batch_size in the forward() For example, the code below was provided by GPT: ...
stayfish's user avatar
0 votes
0 answers
20 views

Is token mask masked in attention of encoders of bert?

I have recently researched on Bert structure. And the paper says we will mask some token at the input in 80%, 10% input be changed and 10% left remained. But I wonder if the mask token in the input be ...
Thành An's user avatar
0 votes
1 answer
33 views

How to get Complexity per Layer, Sequential Operations and Maximum Path Length in CNN architecture?

In the paper Attention is all you need, here is Table 1, can someone explain what architecture is referred to in the "Convolution" row and hence describe the other 3 columns in it? The other ...
Harry's user avatar
  • 11
0 votes
0 answers
43 views

Why does the algorithm in "Self-attention Does Not Need $O(n^{2})$ Memory" require $O(log n)$ memory when $k, v$ pairs are not ordered?

I am reading Self-attention Does Not Need $O(n^{2})$ Memory which proposes an algorithm that requires $O(1)$ memory for one query and $O(log n)$ memory for self-attention, in theory. In practice the ...
Daviiid's user avatar
  • 575
1 vote
0 answers
12 views

Decision Transformer: more than a "trajectory picking" algorithm?

I'm studying the Decision Transformer for some offline reinforcement learning tasks. The basic idea is to collect a huge quantity of data generated by a real experimental device (let's say an arm ...
Dave's user avatar
  • 155
0 votes
0 answers
11 views

How to construct source padding mask for embedded audio?

I'm attempting a music transcription task - similar to speech recognition but with music and notes (string representations) instead of speech audio and sentences. The model consists of a CNN audio ...
jy99's user avatar
  • 1
0 votes
0 answers
34 views

Attention Mechanism: Why don't we just use a simple dot product instead of the Q, K, V matrices?

I am currently learning about Transformers by reading Richard Turner's paper "An Introduction to Transformers". On page 3 of the paper he gave a "naive" approach to build the ...
StockComCat's user avatar
1 vote
1 answer
67 views

why we use learnable positional encoding instead of Sinusoidal positional encoding

In the original paper of transformers they using positional encoding to capture the position of each word in the sentence and for calculate that it using sin and cos ,like shom in the image. In Bert ...
LAILA EL OUEDEGHYRY's user avatar
0 votes
1 answer
85 views

Train my own LLM on a smaller corpus of text?

Would it be possible to train my own LLM on a smaller corpus of text, lets say some coding documentation that I then want to ask questions about using the model? If so, are there any recommended ways ...
Dylan Dijk's user avatar
1 vote
0 answers
17 views

Compare two songs content using Audio Spectogram Transformer

I'm trying to establish a similarity metric between two songs. To do this I'm using the AST model on HuggingFace. This model basically works in a way very similar to a ViT but applied to spectograms ...
user491880's user avatar
0 votes
2 answers
106 views

How the Q,K,V be calculated in multi-head attention

I want to understand the transformer architecture, so I start with self attention and I understand their mechanism, but when I pass to the multi-head attention I find some difficulties like how ...
LAILA EL OUEDEGHYRY's user avatar
0 votes
0 answers
16 views

Problems with understanding instruction fine-tuning

I'm trying to read up on instruction fine-tuning, but I think I have a big misunderstanding. As I understand, instruction datasets typically have 3 components: (a) the instruction (b) the output/...
Christian's user avatar
  • 101
0 votes
0 answers
29 views

Why doesn't my toy transformer model "grok"?

I'm working on reproducing the results by Neel Nanda on teaching a small transformer to perform modular addition: (operand_1+operand_2)%mod_value. The expectation ...
LawlessWalrus's user avatar
0 votes
0 answers
42 views

How to teach Gemma model my mother tongue (Kannada - one of the oldest Indic languages)

I'm interested in teaching the Gemma 2B model my mother tongue (Kannada - one of the oldest Indic languages). The pre-trained model doesn't work well with the mentioned language, so I thought of ...
Swastik's user avatar
  • 101
0 votes
0 answers
33 views

Correctly applying softmax in self attention layer

I'm trying to understand how to apply softmax in self attention layer. Let's say we have Query and Key matrix where the last row is for Paddings In this case Z = Q*K_t would be something like this: ...
Davk9_4's user avatar
1 vote
0 answers
37 views

Is there any standardized notation for drawing neural network diagrams?

Is there any standardized notation for drawing neural network diagrams? For example, for circuits there is a universal set of symbols used to draw different types of circuits why not for neural ...
play's user avatar
  • 33
0 votes
0 answers
10 views

What effect is expected if LoRA is applied after Fine-tuning?

I am currently learning several things about the ASR Transformer model. Recently, I learned LoRA and Adapter. It certainly seems to have an advantage over fine-tuning in general. But here I came up ...
C yp's user avatar
  • 1
0 votes
0 answers
18 views

Dimension of the embedding matrix in a transformer during inference

I have a bit of confusion understanding the dimensions of the input embedding matrix at inference time in transformers. Some sources say that you start with an token and you fill with tokens up to ...
computer eater's user avatar
4 votes
2 answers
1k views

Why different noise in GAN generate different images?

I understand that noise $z$ serves as the input to the generator. Noise $z$ is essentially a vector of random numbers, typically from Gaussian distribution with chosen size of like $100$. However, I ...
user avatar
1 vote
0 answers
82 views

How do transformer-based architectures generate contextual embeddings?

How do transformer-based architectures like Roberta generate contextual embeddings? The articles I've read keep saying that transformer encoders work bidirectionally. Because of self-attention, they ...
user avatar
1 vote
1 answer
63 views

Fine tuning or just feature extraction or both using Roberta?

I'm reading a program that use the pre-trained Roberta model (roberta-base). The code first extracts word embeddings from each caption in the batch, using the last hidden state of the Roberta model. ...
user avatar
0 votes
1 answer
66 views

How do I code so that the embedding output and input share the same weight matrices?

I am trying to implement the Attention is All You Need paper from scratch. The authors mentioned in section 3.4 that "In our model, we share the same weight matrix between the two embedding ...
OneMoreGamble's user avatar
1 vote
0 answers
65 views

AI chat bot that answers by focusing only on 30 textbooks [closed]

I don't even know what I'm looking for and what's the terminology, so here I am asking this question. Background Assume I have 30 textbooks. I want to have an AI chatbot like ChatGPT which answers the ...
Megidd's user avatar
  • 141
0 votes
1 answer
34 views

My small BERT can't even overfit on a sentiment analysis task

I'm trying to train (from scratch) a miniature BERT model on SST2, a simple binary sentiment analysis task with inputs of maybe 5-20 words at a time. As you can see in my code, my approach is a little ...
Jack M's user avatar
  • 262
1 vote
1 answer
104 views

Why do Transformer decoders use masked self attention when producing new tokens?

I've been reading that transformer decoders use masked self attention so that the decoder can't cheat by looking ahead. For example, when predicting the 6th token in the sequence we shouldn't have ...
Kiran Manicka's user avatar
4 votes
1 answer
785 views

How can Transformers handle random sequences?

I have asked ChatGPT the following: Can you concatenate jfef9230rj2mreg90r23ewfrn02eqwdk and 32ir20r3i2ofg90r32kee? And without any error the model produces: ...
killertoge's user avatar
1 vote
1 answer
184 views

Would AlphaZero perform better if made with transformers?

AlphaZero utilized a residual convolutional neural network to estimate move policy and position value. If it was rebuilt today, would it be more efficient and powerful if they used a transformer ...
Ben G's user avatar
  • 123
0 votes
0 answers
18 views

How to structure encoder and decoder input sequences when building transformers model from scratch

I built a transformers model from scratch in PyTorch. I trained it on a novel in the public domain. My sequences are 30 tokens and the first encoder and decoder sequences, for example, are tokens 0-...
matsuo_basho's user avatar
0 votes
1 answer
32 views

Gradually increasing CPU load on using sentence embeddings model with kmeans

I am having a ML based production application, using flask, deployed on GCP server using gunicorn workers. In each incoming request, a text sentence is received. It is using sentence transformers (All-...
racdev's user avatar
  • 1
0 votes
0 answers
30 views

Are video generation also great at next frame video prediction?

If I have a good video generation model like OpenAI's new Sora, will it be capable of doing just as well at next frame video prediction?
JobHunter69's user avatar
1 vote
3 answers
80 views

Does transformers' self-attention mechanism process tokens independently, or entire sequence at a time?

About attention: the Query, Key and Value vectors (before the linear transformations) are just the entire sequence, that is being inputted, or just each token? Chat-GPT nor Youtube didn't give me a ...
CyberLight 64's user avatar
0 votes
0 answers
39 views

Eval loss when fine-tuning in an unsupervised way/pretraining?

I'm fine-tuning the base Mixtral 8x7B model (4-bit quantized) with Lora on my own data, following these guidelines: https://www.stochastic.ai/blog/xfinance-vs-bloomberg-gpt I'm first fine-tuning it in ...
Jon Flynn's user avatar
  • 101
0 votes
1 answer
524 views

Getting started with training local LLM using python [closed]

As I'm completely new to this field, I find it hard to get started given the requirements I have. I'm a bit overwhelmed by all the models and options that are available. Even though it wasn't ...
Jeanluca Scaljeri's user avatar
0 votes
0 answers
180 views

Why feed forward neural network (FFN) in transformer block has a "contract and expand" pattern?

I noticed that in many (every ?) transformer architecture, the FFN (i.e the MLP network at the end of one transformer block) consists of two linear layers (with an activation) where the first layer ...
Lelouch's user avatar
  • 201

1
2 3 4 5
8