Questions tagged [transformer]

For questions related to the transformer, which is a deep machine learning model introduced in 2017 in the paper "Attention Is All You Need", used primarily in the field of natural language processing (NLP).

Filter by
Sorted by
Tagged with
2
votes
0answers
14 views

What part of the Vaswani et al. is the “transformer”?

Which part of this is the transformer? Ok, the caption says the whole thing is the transformer, but that's back in 2017 when the paper was published. My question is about how the community uses the ...
2
votes
0answers
12 views

How to Select Model Parameters for Transformer (Heads, number of layers, etc)

Is there a general guideline on how the Transformer model parameters should be selected, or the range of these parameters that should be included in a hyperparameter sweep? Number of heads Number of ...
0
votes
1answer
34 views

Can an existing transformer model be modified to estimate the next most probable number in a sequence of numbers?

Models based on the transformer architectures (GPT, BERT, etc.) work awesome for NLP tasks including taking an input generated from words and producing probability estimates of the next word as the ...
1
vote
1answer
42 views

Why does GPT-2 Exclude the Transformer Encoder?

After looking into transformers, BERT, and GPT-2, from what I understand, GPT-2 essentially uses only the decoder part of the original transformer architecture and uses masked self-attention that can ...
0
votes
0answers
17 views

Why (not) using pre-processing before using Transformer models?

Regarding the use of pre-processing techniques before using Transformers models, I read this post that apparently says that these measures are not so necessary nor interfere so much in the final ...
1
vote
1answer
59 views

How to construct Transformers to predict multidimensional time series?

There is plenty of information describing Transformers in a lot of detail how to use them for NLP tasks. Transformers can be applied for time series forecasting. See for example "Adversarial ...
0
votes
0answers
17 views

How does BERT answer questions?

I have been trying to understand how the BERT model works. Specifically, I am trying to understand how it picks up answers to questions on a given passage. I have tried following this blog post and, ...
0
votes
0answers
9 views

In transformer, why does decoder shift its input (output sequence actually) by one position

In the translation task, the input-output sequence may not have position-wise relation, it is not word-by-word tanslation. So why do we need to shift the output sequence by one step?
1
vote
1answer
40 views

How are certain machine learning models able to produce variable-length outputs given variable-length inputs?

Most machine learning models, such as multilayer perceptrons, require a fixed-length input and output, but generative (pre-trained) transformers can produce sentences or full articles of variable ...
1
vote
1answer
42 views

How is the transformers' output matrix size arrived at?

In this tensorflow article, the comments in the code say that MHA should output with one of the dimensions being the sequence length of the query/key. However, that means that the second MHA in the ...
0
votes
1answer
27 views

Sentiment analysis does not handle neturals [closed]

I'm writing some financial tools, I've found highly performant models for question and answering but when it comes to sentiment analysis I haven't found anything that good. I'm trying to use ...
1
vote
0answers
26 views

What does the outputlayer of BERT for masked language modelling look like?

In the tutorial BERT – State of the Art Language Model for NLP the masked language modeling pre-training steps are described as follows: In technical terms, the prediction of the output words ...
1
vote
2answers
54 views

What kind of word embedding is used in the original transformer?

I am currently trying to understand transformers. To start, I read Attention Is All You Need and also this tutorial. What makes me wonder is the word embedding used in the model. Is word2vec or GloVe ...
0
votes
0answers
23 views

What is the purpose of “alignment” in the self-attention mechanism of transformers?

I've been reading about transformers & have been having some difficulty understanding the concept of alignment. Based on this article Alignment means matching segments of original text with their ...
1
vote
0answers
21 views

How does the embeddings work in vision transformer from paper?

I get the part from the paper where the image is split into P say 16x16 (smaller images) patches and then you have to ...
0
votes
0answers
35 views

Is using a LSTM, CNN or any other neural network model on top of a Transformer(using hidden states) overkill?

I have recently come across transformers, I am new to Deep Learning. I have seen a paper using CNN and BiLSTM on top of a transformer, the paper uses a transformer(XLM-R) for sentiment analysis in ...
0
votes
0answers
23 views

Is the dimensionality of the query and key tensors in this implementation of the transformer's self-attention correct?

Looking at this tutorial matmul_qk = tf.matmul(q, k, transpose_b=True) # (..., seq_len_q, seq_len_k) is the output of two tensors multiplied of shape: ...
0
votes
1answer
140 views

Transformers: How to use the target mask properly?

I try to apply Transformers to an unusual use case - predict the next user session based on the previous one. A user session is described by a list of events per second, e.g. whether the user watches ...
1
vote
0answers
54 views

Using Reformer model to do NER in Trax

I have been looking at the NER example with Trax in this notebook: https://github.com/google/trax/blob/master/trax/examples/NER_using_Reformer.ipynb However the notebook only gives an example as far ...
1
vote
2answers
87 views

Recent deep learning textbook (i.e. covering at least GANs, LSTM and transformers and attention)

I am searching for an academic (i.e. with maths formulae) textbook which covers (at least) the following: GAN LSTM and transformers (e.g. seq2seq) Attention mechanism The closest match I got is Deep ...
1
vote
0answers
102 views

What is MNLI-(m/mm)?

I came across the term MNLI-(m/mm) in Table 1 of the paper BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. I know what MNLI stands for, i.e. Multi-Genre Natural ...
0
votes
1answer
33 views

Can the attention mechanism improve the performance in the case of short sequences?

I am aware that the attention mechanism can be used to deal with long sequences, where problems related to gradient vanishing and, more generally, representing effectively the whole sequence arise. ...
1
vote
1answer
33 views

In attention models with multiple layers, are weight matrices shared across layers?

In articles that describe neural architectures with multiple attention layers of the same form, are the weight matrices usually the same across the layers? Consider as an example, "Attention is ...
0
votes
2answers
78 views

In the multi-head attention mechanism of the transformer, why do we need both $W_i^Q$ and ${W_i^K}^T$?

In the Attention is all you need paper, on the 4th page, we have equation 1, which describes the self-attention mechanism of the transformer architecture $$ \text { Attention }(Q, K, V)=\operatorname{...
2
votes
0answers
72 views

How to handle long sequences with transformers?

I have a time series sequence with 10 million steps. In step $t$, I have a 400 dimensional feature vector $X_t$ and a scalar value $y_t$ which I want to predict during inference time and I know during ...
3
votes
2answers
90 views

What is different in each head of a multi-head attention mechanism?

I have a difficult time understanding the "multi-head" notion in the original transformer paper. What makes the learning in each head unique? Why doesn't the neural network learn the same ...
1
vote
1answer
184 views

Can we use transformers for audio classification tasks?

Since transformers are good at processing sequential data, can we also use them for audio classification problems (same as RNNs)?
1
vote
0answers
4 views

POS Tag Frequency and Language Translation

When we translate a text from one language to another, how does the frequency of various POS tags change? So let's say we have a text in English with 10% nouns, 20% adjectives, 15% adverbs, 25% verbs, ...
2
votes
0answers
22 views

Why does the loss stops reducing after a point in this Transformer Model?

Context I was making a Transformer Model to convert English Sentences to German Sentences. But the loss stops reducing after some time. Code ...
1
vote
0answers
96 views

What is the gradient of an attention unit?

The paper Attention Is All You Need describes the Transformer architecture, which describes attention as a function of the queries $Q = x W^Q$, keys $K = x W^K$, and values $V = x W^V$: $\text{...
2
votes
1answer
61 views

What is the cost function of a transformer?

The paper Attention Is All You Need describes the transformer architecture that has an encoder and a decoder. However, I wasn't clear on what the cost function to minimize is for such an architecture. ...
1
vote
1answer
25 views

Is the Decoder mask (triangular mask) applied only in the first decoder block, or to all blocks in Decoder?

The Decoder mask, also called "look-ahead mask", is applied in the Decoder side to prevent it from attending future tokens. Something like this: ...
2
votes
1answer
40 views

Transformers: how to get the output (keys and values) of the encoder?

I was reading the paper Attention Is All You Need. It seems like the last step of the encoder is a LayerNorm(relu(WX + B) + X), i.e. an add + normalization. This should result in a $n$ x $d^{model}$ ...
2
votes
1answer
31 views

Transformers: how does the decoder final layer output the desired token?

In the paper Attention Is All You Need, this section confuses me: In our model, we share the same weight matrix between the two embedding layers [in the encoding section] and the pre-softmax linear ...
1
vote
0answers
17 views

How Positional Encoding differs between Transformer and GPT?

I know the original Transformer and the GPT (1-3) use two slightly different positional encoding techniques. More specifically, in GPT they say positional encoding is learned. What does that mean? ...
0
votes
0answers
10 views

Best strategy for Classification of Science Subjects. Phy, Chem , Maths and Bio? BERT, Transformers, Attention+SLTM, Self-Attention+LSTM?

I am working on a project where I have to first classify the Subjects of the given question and then the respective Chapter and then the sub-topic. In a nutshell, I have to predict the Subject, Grade ...
4
votes
1answer
79 views

Why are “Transformers” called this way?

What is the reason behind the name "Transformers", for Multi Head Self-Attention-based neural networks from Attention is All You Need? I have been googling this question for a long time, and ...
0
votes
0answers
22 views

How does scheduled sampling for transformers work?

I was reading this paper which applies a modified version of the transformers for traffic forecasting. I am somewhat familiar with the transformer architecture and how it functions, but, in the paper, ...
0
votes
0answers
20 views

Is there a pretrained (NLP) transformer that uses subword n-gram embeddings for tokenization like fasttext?

I know that several tokenization methods that are used for tranformer models like WordPiece for Bert and BPE for Roberta and others. What I was wondering if there is also a transformer which uses a ...
0
votes
0answers
18 views

What BERT predicts when the token supposed to be masked is not masked?

I am reading the BERT paper. In the paper, they say that: Although this allows us to obtain a bidirec- tional pre-trained model, a downside is that we are creating a mismatch between pre-training and ...
0
votes
0answers
33 views

Bigger models get higher losses

I'm training a model with the transformer encoder architecture on a given fixed set of data. The task I'm solving has a trivial approximation which consists in copying part of the input to the output, ...
0
votes
0answers
21 views

Transformer Language model produces only <pad> tokens when generating new sentences

I am training a word-level language model using the transformer module available in Pytorch. I am getting a really good training loss and the model is able to reproduce the sentences in the training ...
2
votes
1answer
59 views

Any comparison between transformer and RNN+Attention on the same dataset?

I am wondering what is believed to be the reason for superiority of transformer? I see that some people believe because of the attention mechanism used, it’s able to capture much longer dependencies. ...
5
votes
3answers
294 views

What is the purpose of Decoder mask (triangular mask) in Transformer?

I'm trying to implement transformer model using this tutorial. In the decoder block of the Transformer model, a mask is passed to "pad and mask future tokens in the input received by the decoder&...
0
votes
0answers
60 views

Transformer Language Model generating meaningless text

I currently learning on Transformers, so check my understanding I tried implementing a small transformer-based language model and compare it to RNN based language model. Here's the code for ...
1
vote
0answers
54 views

How are weight matrices in attention trained?

I have been looking into transformers lately and been reading tons of tutorials. All of them address the intuition behind attention, which I understand, but they treat training the weight matrices for ...
0
votes
1answer
58 views

How to implement or avoid masking for transformer?

When it comes to using Transformers for image captioning is there any reason to use masking? I currently have a resnet101 encoder and am trying to use the features as the input for a transformer model ...
4
votes
0answers
45 views

Are there transformer-based architectures that can produce fixed-length vector encodings given arbitrary-length text documents?

BERT encodes a piece of text such that each token (usually words) in the input text map to a vector in the encoding of the text. However, this makes the length of the encoding vary as a function of ...
0
votes
1answer
40 views

BERT: After pretraining 880000 step, why fine-tune not work? [closed]

I am using pretraining code from https://github.com/NVIDIA/DeepLearningExamples Pretrain parameters: ...
0
votes
0answers
45 views

How to input a given sequence to a transformer (or an RNN) with probability of occurrence?

I'm experimenting with music and transformers, and I have sequences $S$ of shape: $(B,L,N)$ where $B$ is the batch size, $L$ is the sequence length, and $N=12$ are the number of musical notes with ...