Skip to main content

Questions tagged [transformer]

For questions related to the transformer, which is a deep machine learning model introduced in 2017 in the paper "Attention Is All You Need", used primarily in the field of natural language processing (NLP).

Filter by
Sorted by
Tagged with
0 votes
0 answers
7 views

Is token mask masked in attention of encoders of bert?

I have recently researched on Bert structure. And the paper says we will mask some token at the input in 80%, 10% input be changed and 10% left remained. But I wonder if the mask token in the input be ...
Thành An's user avatar
0 votes
1 answer
25 views

How to get Complexity per Layer, Sequential Operations and Maximum Path Length in CNN architecture?

In the paper Attention is all you need, here is Table 1, can someone explain what architecture is referred to in the "Convolution" row and hence describe the other 3 columns in it? The other ...
Harry's user avatar
  • 1
0 votes
0 answers
14 views

Why does the algorithm in "Self-attention Does Not Need $O(n^{2})$ Memory" require $O(log n)$ memory when $k, v$ pairs are not ordered?

I am reading Self-attention Does Not Need $O(n^{2})$ Memory which proposes an algorithm that requires $O(1)$ memory for one query and $O(log n)$ memory for self-attention, in theory. In practice the ...
Daviiid's user avatar
  • 575
1 vote
0 answers
12 views

Decision Transformer: more than a "trajectory picking" algorithm?

I'm studying the Decision Transformer for some offline reinforcement learning tasks. The basic idea is to collect a huge quantity of data generated by a real experimental device (let's say an arm ...
Dave's user avatar
  • 155
0 votes
0 answers
6 views

How to construct source padding mask for embedded audio?

I'm attempting a music transcription task - similar to speech recognition but with music and notes (string representations) instead of speech audio and sentences. The model consists of a CNN audio ...
jy99's user avatar
  • 1
0 votes
0 answers
27 views

Attention Mechanism: Why don't we just use a simple dot product instead of the Q, K, V matrices?

I am currently learning about Transformers by reading Richard Turner's paper "An Introduction to Transformers". On page 3 of the paper he gave a "naive" approach to build the ...
StockComCat's user avatar
1 vote
1 answer
33 views

why we use learnable positional encoding instead of Sinusoidal positional encoding

In the original paper of transformers they using positional encoding to capture the position of each word in the sentence and for calculate that it using sin and cos ,like shom in the image. In Bert ...
LAILA EL OUEDEGHYRY's user avatar
0 votes
0 answers
36 views

Train my own LLM on a smaller corpus of text?

Would it be possible to train my own LLM on a smaller corpus of text, lets say some coding documentation that I then want to ask questions about using the model? If so, are there any recommended ways ...
Dylan Dijk's user avatar
1 vote
0 answers
14 views

Compare two songs content using Audio Spectogram Transformer

I'm trying to establish a similarity metric between two songs. To do this I'm using the AST model on HuggingFace. This model basically works in a way very similar to a ViT but applied to spectograms ...
user491880's user avatar
0 votes
1 answer
31 views

Literature suggestions for transformers

What are the best educational sources for learning about transformers, what is the go to literature for a mathematician who considers themself a beginner in the subject? Books, lecture notes, research ...
Monty's user avatar
  • 101
0 votes
2 answers
77 views

How the Q,K,V be calculated in multi-head attention

I want to understand the transformer architecture, so I start with self attention and I understand their mechanism, but when I pass to the multi-head attention I find some difficulties like how ...
LAILA EL OUEDEGHYRY's user avatar
0 votes
0 answers
15 views

Problems with understanding instruction fine-tuning

I'm trying to read up on instruction fine-tuning, but I think I have a big misunderstanding. As I understand, instruction datasets typically have 3 components: (a) the instruction (b) the output/...
Christian's user avatar
  • 101
0 votes
0 answers
27 views

Why doesn't my toy transformer model "grok"?

I'm working on reproducing the results by Neel Nanda on teaching a small transformer to perform modular addition: (operand_1+operand_2)%mod_value. The expectation ...
LawlessWalrus's user avatar
0 votes
0 answers
24 views

How to teach Gemma model my mother tongue (Kannada - one of the oldest Indic languages)

I'm interested in teaching the Gemma 2B model my mother tongue (Kannada - one of the oldest Indic languages). The pre-trained model doesn't work well with the mentioned language, so I thought of ...
Swastik's user avatar
0 votes
0 answers
24 views

Correctly applying softmax in self attention layer

I'm trying to understand how to apply softmax in self attention layer. Let's say we have Query and Key matrix where the last row is for Paddings In this case Z = Q*K_t would be something like this: ...
Davk9_4's user avatar
1 vote
0 answers
33 views

Is there any standardized notation for drawing neural network diagrams?

Is there any standardized notation for drawing neural network diagrams? For example, for circuits there is a universal set of symbols used to draw different types of circuits why not for neural ...
play's user avatar
  • 33
0 votes
0 answers
8 views

What effect is expected if LoRA is applied after Fine-tuning?

I am currently learning several things about the ASR Transformer model. Recently, I learned LoRA and Adapter. It certainly seems to have an advantage over fine-tuning in general. But here I came up ...
C yp's user avatar
  • 1
0 votes
0 answers
16 views

Can all good language models be used as a good embedding model and vice versa?

Does solving the language modeling task go hand in hand with finding a good embedding?
JobHunter69's user avatar
0 votes
0 answers
17 views

Dimension of the embedding matrix in a transformer during inference

I have a bit of confusion understanding the dimensions of the input embedding matrix at inference time in transformers. Some sources say that you start with an token and you fill with tokens up to ...
computer eater's user avatar
4 votes
2 answers
1k views

Why different noise in GAN generate different images?

I understand that noise $z$ serves as the input to the generator. Noise $z$ is essentially a vector of random numbers, typically from Gaussian distribution with chosen size of like $100$. However, I ...
abcd's user avatar
  • 153
2 votes
0 answers
81 views

How do transformer-based architectures generate contextual embeddings?

How do transformer-based architectures like Roberta generate contextual embeddings? The articles I've read keep saying that transformer encoders work bidirectionally. Because of self-attention, they ...
abcd's user avatar
  • 153
1 vote
1 answer
37 views

Fine tuning or just feature extraction or both using Roberta?

I'm reading a program that use the pre-trained Roberta model (roberta-base). The code first extracts word embeddings from each caption in the batch, using the last hidden state of the Roberta model. ...
abcd's user avatar
  • 153
0 votes
1 answer
55 views

How do I code so that the embedding output and input share the same weight matrices?

I am trying to implement the Attention is All You Need paper from scratch. The authors mentioned in section 3.4 that "In our model, we share the same weight matrix between the two embedding ...
OneMoreGamble's user avatar
1 vote
0 answers
61 views

AI chat bot that answers by focusing only on 30 textbooks [closed]

I don't even know what I'm looking for and what's the terminology, so here I am asking this question. Background Assume I have 30 textbooks. I want to have an AI chatbot like ChatGPT which answers the ...
Megidd's user avatar
  • 141
0 votes
1 answer
32 views

My small BERT can't even overfit on a sentiment analysis task

I'm trying to train (from scratch) a miniature BERT model on SST2, a simple binary sentiment analysis task with inputs of maybe 5-20 words at a time. As you can see in my code, my approach is a little ...
Jack M's user avatar
  • 262
1 vote
1 answer
83 views

Why do Transformer decoders use masked self attention when producing new tokens?

I've been reading that transformer decoders use masked self attention so that the decoder can't cheat by looking ahead. For example, when predicting the 6th token in the sequence we shouldn't have ...
Kiran Manicka's user avatar
4 votes
1 answer
780 views

How can Transformers handle random sequences?

I have asked ChatGPT the following: Can you concatenate jfef9230rj2mreg90r23ewfrn02eqwdk and 32ir20r3i2ofg90r32kee? And without any error the model produces: ...
killertoge's user avatar
1 vote
1 answer
146 views

Would AlphaZero perform better if made with transformers?

AlphaZero utilized a residual convolutional neural network to estimate move policy and position value. If it was rebuilt today, would it be more efficient and powerful if they used a transformer ...
Ben G's user avatar
  • 123
0 votes
0 answers
16 views

How to structure encoder and decoder input sequences when building transformers model from scratch

I built a transformers model from scratch in PyTorch. I trained it on a novel in the public domain. My sequences are 30 tokens and the first encoder and decoder sequences, for example, are tokens 0-...
matsuo_basho's user avatar
0 votes
1 answer
27 views

Gradually increasing CPU load on using sentence embeddings model with kmeans

I am having a ML based production application, using flask, deployed on GCP server using gunicorn workers. In each incoming request, a text sentence is received. It is using sentence transformers (All-...
racdev's user avatar
  • 1
0 votes
0 answers
25 views

Are video generation also great at next frame video prediction?

If I have a good video generation model like OpenAI's new Sora, will it be capable of doing just as well at next frame video prediction?
JobHunter69's user avatar
1 vote
3 answers
71 views

Does transformers' self-attention mechanism process tokens independently, or entire sequence at a time?

About attention: the Query, Key and Value vectors (before the linear transformations) are just the entire sequence, that is being inputted, or just each token? Chat-GPT nor Youtube didn't give me a ...
CyberLight 64's user avatar
0 votes
0 answers
35 views

Eval loss when fine-tuning in an unsupervised way/pretraining?

I'm fine-tuning the base Mixtral 8x7B model (4-bit quantized) with Lora on my own data, following these guidelines: https://www.stochastic.ai/blog/xfinance-vs-bloomberg-gpt I'm first fine-tuning it in ...
Jon Flynn's user avatar
  • 101
0 votes
1 answer
369 views

Getting started with training local LLM using python [closed]

As I'm completely new to this field, I find it hard to get started given the requirements I have. I'm a bit overwhelmed by all the models and options that are available. Even though it wasn't ...
Jeanluca Scaljeri's user avatar
0 votes
0 answers
97 views

Why feed forward neural network (FFN) in transformer block has a "contract and expand" pattern?

I noticed that in many (every ?) transformer architecture, the FFN (i.e the MLP network at the end of one transformer block) consists of two linear layers (with an activation) where the first layer ...
Lelouch's user avatar
  • 201
2 votes
1 answer
44 views

What does Figure 3 in the BERT paper represent?

The BERT paper has the following diagram (Figure 3): It's captioned "Differences in pre-training model architectures". However, I thought the BERT architecture was just a stack of attention ...
statusfailed's user avatar
1 vote
1 answer
33 views

How come all the multi-headed self-attention layers don't end up learning the same aspect of a natural language?

How come all the multi-headed self-attention layers don't end up learning the same aspect of a natural language? Since we don't dictate ahead of time what the self-attention layers focus on, how do we ...
Tfovid's user avatar
  • 187
0 votes
1 answer
51 views

Handling Variable Output Token Dimensionality in Transformer Decoders During Inference

I'm curious about something in the decoder part of the Transformers architecture. From what I understand, the Keys and Values come from the output of the encoder part of the Transformer. I understand ...
FluidMechanics Potential Flows's user avatar
0 votes
1 answer
31 views

Can positional encodings in transformers be added

Here's a basic GPT2 implementation: ...
Foobar's user avatar
  • 153
0 votes
0 answers
27 views

Where U-Net and Convolutional layers are settled in Stable Diffusion model?

When I read about Stable Diffusion model, they usually talk about adjusting convolution layers or U-Net weights. I believe they both should be related together and the U-Net is the part that accepts ...
best_of_man's user avatar
0 votes
0 answers
12 views

How to train ViT on smaller datasets?

I know ViTs aren't made for small datasets and low resolution. But have you ever reached traditional CNN accuracy using ViT on CIFAR10/100. I have been playing around with ViT on CIFAR10 and 100. But ...
v1998199904's user avatar
0 votes
0 answers
29 views

Any suggestions for transformer finetuning techniques ablation study?

I'm planning to fine tune a 7b parameter model for a research project. I understand the different steps of model fine tuning, namely Supervised fine tuning - where we train model on curated examples ...
kaiser's user avatar
  • 101
1 vote
1 answer
56 views

Is the multi-headed projection matrix in self-attention redundant?

As I understand it, the forward pass for a transformer model looks as follows: x += self_attention(x) x = layernorm(x) x += ffn(x) Breaking that down a bit (excuse ...
Sue Doh Nimh's user avatar
0 votes
1 answer
66 views

Overcoming the quadratic scaling in transformer architecture

Do you know any papers that try to overcome quadratic scaling problems by attending lower dimensional representations in the dimension of tokens? For example, let's say that the input to the ...
Andy Yermakov's user avatar
0 votes
0 answers
23 views

Are K and V values reused in each decoder layer's cross-Attention in the original "Attention is all you need" paper?

I'm working with Transformers and have a question about the encoder-decoder structure. In each decoder layer's cross-attention, are the K and V pairs from the corresponding encoder layer reused for ...
Dennis Yang's user avatar
0 votes
0 answers
30 views

Why are rows of Attention Weights in a Hopfield Transformer the same?

I'm working on building a Hopfield Transformer using the github code from the paper (https://github.com/ml-jku/hopfield-layers/tree/master/hflayers) to forecast a timeseries dataset with 48 variables, ...
Ryan Bose-Roy's user avatar
2 votes
1 answer
49 views

Why is the sinusoidal model classified as absolute positional encoding in some literature?

I am currently reading in depth about positional encodings, and as we know there are two types of positional encodings: Absolute and relative. My question: Why is the sinusoidal model classified as ...
Ali Haider Ahmad's user avatar
0 votes
0 answers
33 views

What am I doing wrong that result in a graph indicating better gradients in non-scaled dot-product attention compared to the scaled version?

I'm trying to visualize how the gradients change as we're increasing $d_{k}$ in the scaled dot-product attention and compare it to its non scaled version but I'm failing to produce a reasonable graph ...
Daviiid's user avatar
  • 575
1 vote
1 answer
38 views

Comparing the performances of GPTs with deep learning in the field of binary files and their related reports

Regarding the case study of a dataset including binary files (containing assembly code) and reports related to each file (the content of the static analysis of the file as well as the analysis of the ...
user16385455's user avatar
0 votes
0 answers
44 views

Do I need residual block in a transformer model if vanishing gradients don't exist?

In this example l’Afrique is x^3 and the attention is being computed for this word A^3(l’Afrique). In the image above Andrew Ng indicates that the word with the biggest wieght in the computation of A^...
Stef's user avatar
  • 95

1
2 3 4 5
8