Last call to make your voice heard! Our 2022 Developer Survey closes in less than a week. Take survey.

Questions tagged [transformer]

For questions related to the transformer, which is a deep machine learning model introduced in 2017 in the paper "Attention Is All You Need", used primarily in the field of natural language processing (NLP).

Filter by
Sorted by
Tagged with
0 votes
0 answers
19 views

Which index of the output in Transformers is used during inference to predict multiple words?

I am somewhat confused about how transformers, not just the original model, but also models like GPT-2 work when they are not training but are used multiple times to predict single tokens/words. The ...
user avatar
2 votes
1 answer
34 views

When exactly does the split into different heads in Multi-Head-Attention occur?

I am confused by the Multi-Head part of the Multi-Head-Attention used in Transformers. My question concerns the implementations in Pytorch of nn.MultiheadAttention and its forward method ...
user avatar
1 vote
1 answer
36 views

Positional Encoding of Time-Series features

I’m trying to use a Transformer Encoder I coded with weather feature vectors which are basically 11 features about the weather in the dimension ...
user avatar
0 votes
0 answers
22 views

Why are separate, bigger Encoder-Decoder architectures used instead of Bidirectional RNNs/Transformers for Seq2Seq tasks?

Whether with RNNs or Transformers, Encoder-Decoder networks are used for Sequence to Sequence (Seq2Seq) tasks, like Machine Translation. Why are separate, bigger Encoder-Decoder networks used for this ...
user avatar
0 votes
0 answers
22 views

What does "position" in "each position in the decoder" denote in the Transformer's original paper?

I am reading Attention is All You Need and I feel confused about the word "position" in this paper, by the way I'm not native English speaker which may cause my confusion which has confused ...
user avatar
  • 1
1 vote
0 answers
13 views

When training a seq2seq model is it better to train using the models outputs or expected outputs?

When training any seq2seq model you have a target and a source. The source may be a sentence ...
user avatar
  • 1,324
0 votes
1 answer
20 views

Attention: Isn't it redundant to apply a linear layer to both the keys and values?

Transformer attention is calculated $Attention(X) =X W^V\times \text{columnwise-softmax} (Att(X))$ where the attention attention matrix is $$Att(X) = Q \times K = {X W}^Q \times ({X W}^K)^T = {X W}^Q (...
user avatar
0 votes
0 answers
45 views

Does the transformer model have any inherent ordering?

I'm aware of the practice of positional encoding, which inserts information to convey the relative positions of input data points. Unfortunately, I do not have a great grasp of the transformer model ...
user avatar
0 votes
0 answers
14 views

Why is it common to use K=V in Attention layers?

Context: Hi I recently read from the keras docs: "key: Optional key Tensor of shape (B, S, dim). If not given, will use value for both key and value, which is the most common case." I found ...
user avatar
  • 266
1 vote
1 answer
62 views

How to use structural information in a Transformer?

I am performing a Neural Machine Translation (NMT) task. In my case, input data has relational information. I know I can use a Graph Neural Network (GNN) and use a Graph2Seq model. But I can't find a ...
user avatar
0 votes
0 answers
41 views

How to include additional numeric input in the Transformer architecture

I want to apply the Transformer architecture to my machine translation task, and provide the decoder with an additional parameter in the range of [0,1]. This ...
user avatar
0 votes
0 answers
12 views

Transformers for regression on permutation of fixed size sequence?

Transformers have shown remarkable performance operating on sequences, but are equivariant to the order in the input sequence. Positional Encoding alleviates that problem, but how good is it? In my ...
user avatar
  • 246
0 votes
0 answers
9 views

CTC Loss incredibly low for wrong output

I am trying to train an OCR model with Vision Transformers. While training the output is a vector with values full of zeros which is obviously padding value. But the CTC loss was small that it was ...
user avatar
0 votes
0 answers
16 views

What is the canocial way to handling differing input and output dimensions for the transformer model?

I have an essential regression task, where the input is of dimension $d$ and the output is a scalar. I think the transformer model is a good fit for this problem. In the vanilla multi-head-attention ...
user avatar
  • 41
0 votes
0 answers
14 views

How to interpret Transformer output

In this (https://towardsdatascience.com/a-detailed-guide-to-pytorchs-nn-transformer-module-c80afbc9ffb1) article the author says, that the output of the ...
user avatar
0 votes
1 answer
64 views

Dimensions of a Transformer model and purpose of masking [closed]

I'm currently studying the Transformer model (Attention is all you need) and after reading it I still have some questions for which I get conflicting answers if I google them: What exactly are the ...
user avatar
0 votes
1 answer
97 views

How "Patch Merging" works in SWIN-Transformers?

In the SOTA paper: SWIN-Transformers, the authors have tried their best to explain everything clearly. I have got an idea of how it works except the Patch Merging ...
user avatar
  • 233
0 votes
0 answers
134 views

How do we reduce the output dimensions of BERT?

The output dimensions of BERT are 768-dimensional, is it possible to reduce them to a lower, custom number? For example, if there is another BERT-based transformer model which has a lower count of ...
user avatar
0 votes
1 answer
21 views

Are positional embeddings computed during or before training?

I'm trying to practically frame the concept of positional embeddings as introduced in the original paper. As far as I've understood, what we do is basically creating some other vectors in addition to ...
user avatar
2 votes
1 answer
90 views

What is multi-head attention doing mathematically, and how is it different from self-attention?

I'm trying to understand the difference between the concept of self-attention and multi-head attention. The latter is not actually too clear to me. I understand that, in the case of self-attention, we ...
user avatar
0 votes
1 answer
103 views

Is a Transformer a good choice for multivariate signal classification?

I am working on a problem regarding the multi-classification of multivariate time signals. So I have multiple signals and try to train an algorithm on them. My current approach is to build a neural ...
user avatar
0 votes
2 answers
83 views

Are there any works that deal with 2D pose estimation in videos?

Since pose estimation is often a task where spatial-temporal context should be helpful in finding subsequent key points, I thought there should be many papers on it. However, I could not find any work ...
user avatar
0 votes
0 answers
41 views

Are there any inference memory requirement tables for Hugging Face transformers?

Hugging Face has a very large list of supported transformers. They provide a table which gives the status on whether or not a transformer has a slow tokenizer, a fast tokenizer, PyTorch support, ...
user avatar
0 votes
0 answers
34 views

Why is BERT/GPT capable of "for-all" generalization?

As shown in the figure: Why does token prediction work when "Socrates" is replaced with "Plato"? From the point of view of symbolic logic, the above example effectively performs ...
user avatar
2 votes
2 answers
201 views

Why do Transformers have a sequence limit at inference time?

As far as I understand, Transformer's time complexity increases quadratically with respect to the sequence length. As a result, during training to make training feasible, a maximum sequence limit is ...
user avatar
0 votes
1 answer
22 views

What does it mean to apply decomposition at inference-time in a machine translation system?

I'm reading this paper for sub-character decomposition for logographic languages and the authors mention decomposition at inference-time. They're using Transformer architecture. More specifically, the ...
user avatar
0 votes
1 answer
69 views

Is Positional Encoding always needed for using Transformer models correctly?

I am trying to make a model that uses a Transformer to see the relationship between several data vectors, but the order of the data is not relevant in this case, so I am not using the Positional ...
user avatar
  • 171
0 votes
1 answer
58 views

Why do language models produce different outputs for same prompt?

For conventional 'Neural Networks', the weights simply act as a transformation in highly multi-dimensional space; for a forward pass, the output is always the same since there is no stochastic ...
user avatar
  • 126
0 votes
0 answers
22 views

Why do we run $QK^T$ in self-attention when it can be simplified?

$$ Q = \pmb x W^Q \\ V = \pmb x W^V $$ So $$ \begin{align*}\\ QV^T &= \pmb x W^Q (\pmb x W^V)^T \\ &= \pmb x W^Q(W^V)^T \pmb x^T \\ &= \pmb x M \pmb x^T \end{align*} $$ So you ...
user avatar
0 votes
1 answer
245 views

What is input (and shape) to K/V/Q of self-attention of EACH Decoder block of Language-translation model Transformer's tokens during Inference?

Transformer model of the original Attention paper has a decoder unit that works differently during Inference than Tranining. I'm trying to understand the shapes used during decoder (both self-...
user avatar
  • 151
1 vote
0 answers
97 views

Where do the characteristics of self-attention come into play in Linformer's proof that self-attention is low rank?

In Linformer's proof that self-attention is low rank in their paper, I don't see how it doesn't generalize to every matrix. They don't utilize any specifics of self-attention (the entire proof feels ...
user avatar
  • 2,239
1 vote
1 answer
126 views

Do Vision Transformers handle arbitrary sequence lengths the same way as normal Transformers?

Does ViT do handle arbitrary sequence lengths using masking the same way the normal Transformer does? The ViT paper doesn't mention anything about it, so I assume it uses masking like the normal ...
user avatar
  • 13
2 votes
1 answer
31 views

Is there a notion of location in Transformer architecture in subsequent self-attention layers?

Transformer architecture (without position embedding) is by the very construction equivariant to the permutation of tokens. Given query $Q \in \mathbb{R}^{n \times d}$ and keys $K \in \mathbb{R}^{n \...
user avatar
3 votes
1 answer
142 views

What is the Intermediate (dense) layer in between attention-output and encoder-output dense layers within transformer block in PyTorch implementation?

In PyTorch, transformer (BERT) models have an intermediate dense layer in between attention and output layers whereas the BERT and Transformer papers just mention the attention connected directly to ...
user avatar
1 vote
0 answers
59 views

Is there any point in adding the position embedding to the class token in Transformers?

The popular implementations of ViTs by Ross Wightman and Phil Wang add the position embedding to the class tokens as well as to the patches. Is there any point in doing so? The purpose of introduction ...
user avatar
2 votes
1 answer
78 views

What are the major layers in a Vision Transformer?

Currently, I am studying deepfake detection using deep learning methods. Convolution neural networks, recurrent neural networks, long-short term memory networks, and vision transformers are famous ...
user avatar
1 vote
0 answers
32 views

Why does research on faster Transformers focus on the query-key product?

A lot of recent research on Transformers has been devoted to reducing the cost of the self-attention mechanism: $$\text{softmax}\left(\frac{Q K^T}{\sqrt{d}} \right)V,$$ As I understand it, the runtime,...
user avatar
3 votes
2 answers
201 views

In layman terms, what does "attention" do in a transformer?

I heard from many people about the paper titled Attention Is All You Need by Ashish Vaswani et al. What actually does the "attention" do in simple terms? Is it a function, property, or some ...
user avatar
  • 3,059
1 vote
1 answer
48 views

Can I use the transformers for the prediction of historical data?

Can I use the transformers for the prediction of wind power with the historical data? Dataset Datetime, Ambient temperature (Degree), Dewpoint (Degree), Relative Humidity\n (%), Air Pressure, Wind ...
user avatar
1 vote
0 answers
40 views

What is the purpose of hard distillation?

In order to get a smaller model, one often uses larger model, that performs reasonably well on the data as a teacher, and uses the information from large model to train the smaller one. There are ...
user avatar
0 votes
0 answers
27 views

Is the multi-head attention in the transformer a weighted adjacency matrix?

Are multi-head attention matrices weighted adjacency matrices? The job of the multi-head-attention mechanism in transformer models is to determine how likely a word is to appear after another word. In ...
user avatar
0 votes
0 answers
16 views

What are the pros and cons of using a normal positional encoding in an adjacency matrix?

I understand that a normal positional encoding helps a transformer to understand pictures better and that it allows the (otherwise permutational invariant transformer-network) to create relationships ...
user avatar
0 votes
0 answers
17 views

What is the difference between supervised and unsupervised training in T5?

I know unsupervised training for T5 is like: input: He went X output: X to school Z is this equivalent to the following in a supervised manner: ...
user avatar
  • 101
2 votes
1 answer
76 views

How to generate a response while considering past questions as well?

User: What is the tallest mountain? Agent: Everest User: Where is it located? # Agent hears: "Where is Everest located?" Agent: Nepal I want to be able ...
user avatar
1 vote
2 answers
69 views

Why do the authors of the T5 paper say that the "architectural changes are orthogonal to the experimental factors"?

Here's a quote from the T5 paper (T5 stands for "Text-to-Text Transfer Transformer") titled Exploring the Limits of Transfer Learning with a Unified Text-...
user avatar
0 votes
0 answers
23 views

One single-batch training on Huggingface Bert model "ruins" the model

For some reason, I need to do further (2nd-stage) pre-training on Huggingface Bert model, and I find my training outcome is very bad. After debugging for hours, surprisingly, I find even training one ...
user avatar
  • 101
1 vote
0 answers
84 views

Is there a proper initialization technique for the weight matrices in multi-head attention?

Self-attention layers have 4 learnable tensors (in the vanilla formulation): Query matrix $W_Q$ Key matrix $W_K$ Value matrix $W_V$ Output matrix $W_O$ Nice illustration from https://jalammar....
user avatar
0 votes
1 answer
306 views

Transformer model is very slow and doesn't predict well

I created my first transformer model, after having worked so far with LSTMs. I created it for multivariate time series predictions - I have 10 different meteorological features (temperature, humidity, ...
user avatar
  • 103
2 votes
1 answer
520 views

Why does a transformer not use an activation function following the multi-head attention layer?

I was hoping someone could explain to me why in the transformer model from the "Attention is all you need" paper there is no activation applied after both the multihead attention layer and ...
user avatar
1 vote
1 answer
39 views

How is Google Translate able to translate texts of arbitrarily large length?

Sequence-to-sequence models with attention are known to be limited by a maximum sequence length. So how can we handle sequences of arbitrarily large size? Do we just set a very large maximum sequence ...
user avatar
  • 71