Questions tagged [transformer]
For questions related to the transformer, which is a deep machine learning model introduced in 2017 in the paper "Attention Is All You Need", used primarily in the field of natural language processing (NLP).
231
questions
2
votes
0
answers
64
views
How code analysis part of ChatGPT works and trained
ChatGPT can explain given code snippet we also ask question like "What does this variable do" , "Why this is used" and all. I gave C++ function snippet from an popular Open Source ...
- 121
1
vote
1
answer
103
views
How does transformer models like GPT generate valid meaningful response for meaningless garbage input?
My understanding of a transformer model is that it uses the given input to calculate internal query of relate-ness of word meanings, and generate a meaningful response based on its meaning. But if ...
- 55
0
votes
1
answer
28
views
Going beyond intent classification
I've been working on text -> intent -> command execution for a particular application and while I've found many papers and code that work well for intent classification (1, 2, etc.), they stop ...
- 101
0
votes
1
answer
44
views
Where to find the source code for the research paper "Attention is all you need"?
I am reading "Attention is all you need". I have seen the following link:
Source: https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
But when I go to ...
0
votes
0
answers
5
views
is the length of output of the intermediate decoder of transformer diff between different decoding round in inference mode?
as we know that during inference mode the decoder in transformer generate one word per round/step. the process is like:
step-1 input: <start>
step-1 output: w1
step-2 input: <start> w1
...
- 101
0
votes
1
answer
31
views
Grading student answers using sentence-transformer
I have a dataset that has questions, student answers, reference answers, and a score(0-4).
I want to use sbert to grade student answers by considering question, student answer and reference answer.
...
- 23
0
votes
1
answer
23
views
Fine-Tuning T5 with specific penalty
Currently I am finetuning transformers T5 model for translation task. As part of the dataset, I am given sentences in Japanese, their translation to English, and ...
- 121
1
vote
0
answers
14
views
Is the input embedding split along the embedding dimension so that every head of the multi-head-attention module just gets a part of the input data?
So I found two contradictory explanations of the MHA (multi-head-self-attention-module):
In the first approach, the input embedding (= the input matrix) is split along the embedding dimension and all ...
- 11
0
votes
0
answers
29
views
Why can't traditional neural networks learn to perform the same tasks that attention layers do?
If your task is to predict $t_{n+1}$ given tokens $(t_1,...,t_n)$, you could do two things:
Straight NN - feed $t=(t_1,...,t_n)$ into a neural network as an n-dimensional input and train it on ...
0
votes
0
answers
17
views
Can I use a pre-trained BERT to generate embeddings for training dataset then to fine tune the same BERT for semantic similarity?
I would like to fine-tune a sentence BERT model using my own dataset and perform a semantic similarity task. When generating the training dataset, I need to generate the embeddings for each sentence ...
- 101
1
vote
0
answers
31
views
Using root and modifiers in translation task
I am doing a project of translation task by finetuning T5 model. I am given sentences in Chinese, their translation to English, and for every English sentence I am ...
- 121
0
votes
0
answers
21
views
is mask needed for transformer decoder during inference mode [duplicate]
I know that mask is needed for decoder self-attention during training mode because it can prevent attending "future input tokens".
however, in inference mode, input is generated from ...
- 101
0
votes
0
answers
14
views
Transformer parallelization during training
What does it mean that the decoder can be parallelized during training?
Let's assume a transformer (with both encoder and decoder) is employed for a time-series prediction. I.e. from the input ...
- 111
0
votes
0
answers
25
views
What's the Time and Space Complexity of Transformer Models Inference? [closed]
What's the Big (O) at inference time for transformer models? Is it different for BERT? RoBERTa? T5? DeBERTa?
1
vote
0
answers
19
views
Why does the pass@k metric not "behave like" probability?
pass@k is a metric used to evaluate models that generate code, used for example to evaluate Codex. To evaluate pass@k, you have a dataset of natural language/code pairs, and you pass each NL prompt to ...
- 201
0
votes
0
answers
18
views
How much do we know about the architectures of the Codex (prototype) models?
The transformer model Codex by OpenAI was introduced in a 2021 paper. The paper does not give complete information about the architecture. Below I've quoted all the passages in the paper that give ...
- 201
0
votes
0
answers
22
views
How to use a framework for NLP such as transformers with pytorch or tensorflow to generate statistics based on prompt messages about a dataset?
Is there a way to achieve this or do I have to go in other direction?
It should answer questions such as which order in the last year took the longest time to complete and why.
For instance, with a ...
- 101
1
vote
0
answers
67
views
Coding a conversational AI which remembers previous context
I am trying to code a proper conversational AI which remembers previous context and answers accordingly (something like a micro ChatGPT). Additionally I want the AI to work on a custom knowledge base ...
1
vote
1
answer
72
views
Transformers: how does stacking work? [closed]
An Encoder has as inputs : Q,K,V, but has single output i.e. 3 vs 1
How do you stack those ?
Is there more detailed diagram ?
- 113
1
vote
1
answer
41
views
Are there versions of attention that do not require a key-value pair, but just act on one input?
Are there versions of attention that do not require a key-value pair, but just act on one input? Or does this idea simply not make sense?
1
vote
1
answer
31
views
Why is it called a Seq2Seq model if the output is just a number?
Why is it called a Seq2Seq model if the output is just a number?
For example, if you are trying to predict a movie's recommendation, and you are inputting a ...
- 151
1
vote
0
answers
60
views
Latent Diffusion Model Can't Learn the Latent Space of a VAE for the MNIST-Fashion Dataset
I'm currently playing around with LDMs on the MNIST-Fashion dataset. I thought the VQVAEs used in the original paper were a bit overkill for what I'm doing (and I don't fully understand how they ...
- 127
0
votes
0
answers
13
views
Does attention in transformers encode any information from positional embeddings?
I know we account for positional embeddings before feeding into attention layers, but would we be able to say that the Q and K dot products intrinsically encode relative positions
- 1
0
votes
0
answers
34
views
Why do LLMs like GPT-3 or Bloom use Vanilla Transformer instead of long sequence variants like Transformer-XL?
Is there any particular reason that the most recent and successful large language models like GPT-3 or Bloom utilize a vanilla Transformer architecture instead of an arguably superior long sequence ...
0
votes
0
answers
8
views
How many pretraining image is enough for Swin Transformer?
Here is the spec of experiment setup:
We have 3D micro CT image of the rats, and we want to perform pretraining on such data. The image is masked, so only the portion around the backbone is visible.
...
5
votes
1
answer
122
views
"Attention is all you need" paper : How are the Q, K, V values calculated?
The seminal Attention is all you need paper introduces Transformers and implements the attention mecanism with "queries, keys, values", in an analogy to a retrieval system.
I understand the ...
- 191
0
votes
1
answer
38
views
How do they make transformers bigger/deeper?
I can find a million explanations of the diagram in the original transformer paper:
But I know that modern GPT models have many millions of weights. Where are they? Or in other words, how does this ...
- 121
0
votes
1
answer
126
views
Why does CLIP use a decoder-only transformer for encoding text?
In CLIP [1], the authors train a model to learn multi-modal (text, audio) embeddings by maximizing the cosine similarity between text and image embeddings produced by text and image encoders.
For the ...
- 251
0
votes
0
answers
80
views
When multiple stacked encoders are used, do the decoders only attend to the output of the final encoder layer?
After looking into transformer-based models that used multiple stacked encoders and decoders, I am trying to understand how cross attention in the decoders work. In a transformer with a single encoder/...
- 1
1
vote
0
answers
26
views
Computation required for GPT model to choose likely word from n-options where n < total vocabulary size
Let’s imagine two different use cases for a LLM/GPT-3.
Predicting the next most likely word in a sequence using all ~50k words in its dictionary (i.e. the standard method of prompting a LLM)
Checking ...
- 11
0
votes
0
answers
18
views
Is a GPU needed for Inference in a few-shot setting
In the few-shot setting, it is common to train the model using episodes consisting of a support set and a query set. The way I understand this is, that at the training stage the model is fed with a ...
- 1
0
votes
0
answers
36
views
Is there any evidence that the bias terms thelp in the attention mechanism of the transformers?
In the original transformer paper, the attention mechanism uses parameter matrices, but no bias terms. However, in more recent implementations I see people often using a bias term when computing "...
- 1
0
votes
0
answers
26
views
"Following instructions" as an emergent behaviour in transformer models - isn't this fundamentally different from the models' basic purpose?
I am not technically familiar with AI or neural networks beyond a tech news reading level of knowledge, so I apologise if this is a dumb question.
I was recently reading this article on Ars Technica. ...
- 101
2
votes
1
answer
71
views
Would a transformer trained on highly specific material be as usable as a commercial product like ChatGPT?
Soft question here.
I was recently learning a bit about how it is feasible to train a transformer on a personal computer like an M1 Mac. I have been told that the model could have 1-3 million ...
- 119
0
votes
0
answers
12
views
Can attention models learn statistical parameters such as median, max, mode, mean?
We have some mixed models where after predicting the next word of a sequence, we also want to predict some weights associated to it, related to previous weights of previous words. As the prediction of ...
- 51
0
votes
1
answer
533
views
How to assess if OpenAI's ChatGPT chatbot has a human in the loop? [closed]
I've asked a question and given a couple answers that propose the OpenAI ChatGPT chatbot has humans in the loop (HITL), and that explains the chatbot's extraordinary abilities.
I've been repeatedly ...
- 337
0
votes
1
answer
31
views
Is $i$ indexing the first or second dimension in $\mathbf{x}_i$, where $\mathbf{x} \in \mathbb{R}^{n\times d}$?
I was reading the following notes on the math behind transformers and was confused about what $\mathbf{x}_i$ is? If $\mathbf{x} \in \mathbb{R}^{n\times d}$, then is the $i$ indexing the $n$ or the the ...
- 23
3
votes
2
answers
76
views
Does the position of the tokens in Vision Transformer matter?
I am reading through the Vision Transformer paper and other related papers, such as DeiT and Visual Prompt Tuning (VPT). I wonder if the position of the tokens that flow through the Transformer encode ...
- 1,007
0
votes
0
answers
43
views
How do transformers handle multidimensional input?
Transformers work with lists of vectors, i.e. sentence of length SEQ_LEN, with each word having size EMBEDDING_DIM. Now, since the model still makes use of Dense layers internally, i.e. as in https://...
1
vote
1
answer
27
views
Explanation of Cross-Modality Linear Transformer
So I am trying to understand how a Cross-Modality Linear Transformer is different from an a basic transformer. I found the transformer mentioned in this paper. Am I correct in understanding that, the ...
0
votes
0
answers
18
views
Which pretrained model would best suite for the probelm of generating questions (MCQ) from text?
Here is my use case, I give my model the following text
I brought a car because it is cheap
output is
...
- 208
0
votes
0
answers
18
views
Tokenization for treelike structures
I'm pretraining a BERT (bigbird) model to use with SMILES encoding of chemicals. This kind of data is a treelike structure in the form of a string with a single bracket type. Usually this tree isn't ...
3
votes
1
answer
412
views
What's the relationship between number of heads and embedding dimension in Transformers?
I am reading the book: Natural Language Processing with Transformers. It has the following paragraph
Although head_dim does not have to be smaller than the number of embedding dimensions of the ...
- 480
0
votes
0
answers
53
views
NLP: Question answer with 2 contexts
Is there a Hugging Face Transformer that takes 2 contexts as input for question answering? For example, I could have transcript of a meeting in first context and agenda of the meeting in the second ...
- 31
0
votes
1
answer
117
views
Fine Tuning Transformer Model for Machine Translation
I am working on the Transformer example demonstrated on TensorFlow's website. https://www.tensorflow.org/text/tutorials/transformer
In this example, Machine Translation model is trained to translate ...
- 101
0
votes
2
answers
41
views
Why do the values in the cross attentional mechanism within a transformer come from the encoder and not from the decoder?
The transformer architecture contains a cross attention mechanism which is enriching the encoder with information from the decoder. The place where this takes place is visualized in the image below:
...
- 373
0
votes
0
answers
20
views
How is padding masking considered in the Attention Head of a Transformer?
For purely educational purposes, my goal is to implement basic Transformer architecture from scratch. So far I focused on the encoder for classification tasks and assumed that all samples in a batch ...
- 101
1
vote
1
answer
43
views
What type of neural network architecture allows filtering out of unwanted sounds?
I have a use case where I will be inputting audio to a model, and the output of the model will be the same audio except with certain sounds removed (volume set to zero). The dataset is generated by ...
- 111
1
vote
0
answers
30
views
How do transformers compare to CNNs in terms of compute budget (and computing time) during inference?
Transformers are data and GPU hungry during training. Is this also true at inference time? How do transformers compare to feedforward CNNs e.g., during bounding box generation at inference time? I ...
- 373
3
votes
1
answer
218
views
What is the intuition behind self-attention?
I've been watching a few lectures on transformers, especially for language translation, though it seemingly becomes more confusing the more I watch.
In this lecture, there seems to be two conflicting ...
- 133