Questions tagged [transformer]

For questions related to the transformer, which is a deep machine learning model introduced in 2017 in the paper "Attention Is All You Need", used primarily in the field of natural language processing (NLP).

Filter by
Sorted by
Tagged with
0
votes
1answer
15 views

Why people always say the Transformer is parallelizable while the self-attention layer still depends on outputs of all time steps to calculate?

When compared to an RNN seq-to-seq model, people always say the Transformer is parallelizable. In the original Attention Is All You Need paper, it also said that Recurrent models typically factor ...
0
votes
0answers
13 views

Positional Encoding in Transformer on multi-variate time series data hurts performance

I set up a transformer model that embeds positional encodings in the encoder. The data is multi-variate time series-based data. As I just experiment with the positional encoding portion of the code I ...
2
votes
1answer
42 views

In Computer Vision, what is the difference between a transformer and attention?

Having been studying computer vision for a while, I still cannot understand what the difference between a transformer and attention is?
0
votes
0answers
12 views

Computing the mean attention distance for ViT

Recently I came across the paper that introduces the Vision Transformer (ViT) "AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE". The thing I don't really ...
1
vote
1answer
18 views

Continuous sequence data with Transformer model

What is the right way to input continuous, temporal(time series) data into Transformer network. Assume we're using the basic TransformerBlock here. Since data is continuous with no tokens, Token ...
0
votes
0answers
20 views

Keras MLP performing better than Transformers

I'm working on a comparative study using some models in a sentiment analysis task: MLPs and LSTMs with and without the use of word embeddings (GloVe and Word2Vec) and two Transformer-based models (...
0
votes
0answers
7 views

Can I use transfer learning with Bert and none text sequential data?

I'm working on a multiclass classification problem, each row on my dataset have 5 time windows with 23 values on each time window, I would like to use transfer learning using the Bert transformer to ...
1
vote
0answers
16 views

Why class embedding token is added to the Visual Transformer?

In the famous work on the Visual Transformers, the image is split into patches of a certain size (say 16x16), and these patches are treated as tokens in the NLP tasks. In order to perform ...
0
votes
0answers
24 views

Why and when transformers are better than CNN's in sequence modeling tasks?

Transformers have made a revolution in the domain of NLP and gave rise to a rapid boost of neural networks in a variety of language modelling problems, TTS and, recently, achieved competitive accuracy ...
0
votes
0answers
29 views

What is the best way to generate German paraphrases?

What is the best method to generate German paraphrases? The state-of-the-art are seq2seq transformer models, like T5, but they only work for English sentences. I found the multilingual MT5 model, but ...
0
votes
0answers
12 views

Does the transformers model (in “Attention is All You Need”) exclude the encoder in language modelling tasks?

The language model I am referring to is the one outlined in "Attention is All You Need": My understanding is that when the task is translation, the encoder's input could be "Hi, my ...
0
votes
0answers
11 views

MBART and better Domain Specific Translations Using Masks?

I'm implementing https://huggingface.co/facebook/mbart-large-50-many-to-many-mmt for translations as it has shown promising results but I wanted to see if there was a way to translate specific parts ...
0
votes
0answers
35 views

How do autoregressive attention mechanism work in multi-headed attention?

[LONG POST!!] I am working on a DNN model that works as an improviser to generate music sequences. The idea of generating music is based on taking a sequence of music nodes (their index representation)...
0
votes
0answers
23 views

Positional encoding in convolutional layers

Positional encoding (PE) is an essential part of the self-attention layers in the transformer architectures since without adding it in some way (fixed of learnable) to the input embeddings model has ...
0
votes
0answers
11 views

Visualizing encoder-attention after ResNet in terms of ResNet input

I have a transform-encoder only architecture, which has the following structure: ...
0
votes
1answer
35 views

What makes a transformer a transformer?

Transformers are modified heavily in recent research. But what exactly makes a transformer a transformer? What is the core part of a transformer? Is it the self-attention or the parallelism or ...
2
votes
1answer
66 views

Is it realistic to train a transformer-based model (e.g. GPT) in a self-supervised way directly on the Mel spectrogram?

In music information retrieval, one usually converts an audio signal into some kind "sequence of frequency-vectors", such as STFT or Mel-spectrogram. I'm wondering if it is a good idea to ...
0
votes
0answers
10 views

How does translation of a text in an HTML tag works? Consider the following cases

Recently I came a cross to the Google Translate's add-on for Chrome. Basically you have a web page and it translates that web page for you while keeping its structure. I came to the conclusion that it ...
1
vote
0answers
20 views

Are there any successful applications of transformers of small size (<10k weights)?

In the problems of NLP and sequence modeling Transformer architectures based on self-attention mechanism https://arxiv.org/abs/1706.03762 have achieved impressive results and now are the first choices ...
0
votes
0answers
44 views

Is it sensible to combine GRU/LSTM with the transformer's encoder?

Is it sensible to combine GRU/LSTM with the transformer's encoder? If we take the output of a GRU (uni or bi-directional), and then feed it as input to the transformer's encoder, would that help in ...
1
vote
1answer
50 views

How do transformers understand data and answer custom questions?

I recently heard of GPT-3 and I don't understand how the attention models and transformers encoders and decoders work. I heard that GPT-3 can make a website from a description and write perfectly ...
0
votes
0answers
24 views

Representing variable-length sequences

I want to train a model over a variable-length sequential data (e.g. the temperature at different times of day) where the output depends on what the temperature is at time ...
2
votes
0answers
15 views

Embedding from Transformer-based model from paragraph or documnet (like Doc2Vec)

I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this: ...
0
votes
0answers
15 views

Can transformers be used to improve regression?

I was recently reading a bit about transformers and I don't understand them very much but I was wondering if anyone knows if any of their techniques such as attention mechanism or anything has been ...
2
votes
0answers
19 views

What part of the Vaswani et al. is the “transformer”?

Which part of this is the transformer? Ok, the caption says the whole thing is the transformer, but that's back in 2017 when the paper was published. My question is about how the community uses the ...
2
votes
0answers
16 views

How to Select Model Parameters for Transformer (Heads, number of layers, etc)

Is there a general guideline on how the Transformer model parameters should be selected, or the range of these parameters that should be included in a hyperparameter sweep? Number of heads Number of ...
0
votes
1answer
38 views

Can an existing transformer model be modified to estimate the next most probable number in a sequence of numbers?

Models based on the transformer architectures (GPT, BERT, etc.) work awesome for NLP tasks including taking an input generated from words and producing probability estimates of the next word as the ...
2
votes
2answers
114 views

Why does GPT-2 Exclude the Transformer Encoder?

After looking into transformers, BERT, and GPT-2, from what I understand, GPT-2 essentially uses only the decoder part of the original transformer architecture and uses masked self-attention that can ...
0
votes
0answers
20 views

Why (not) using pre-processing before using Transformer models?

Regarding the use of pre-processing techniques before using Transformers models, I read this post that apparently says that these measures are not so necessary nor interfere so much in the final ...
2
votes
1answer
363 views

How to construct Transformers to predict multidimensional time series?

There is plenty of information describing Transformers in a lot of detail how to use them for NLP tasks. Transformers can be applied for time series forecasting. See for example "Adversarial ...
0
votes
0answers
19 views

How does BERT answer questions?

I have been trying to understand how the BERT model works. Specifically, I am trying to understand how it picks up answers to questions on a given passage. I have tried following this blog post and, ...
0
votes
0answers
9 views

In transformer, why does decoder shift its input (output sequence actually) by one position

In the translation task, the input-output sequence may not have position-wise relation, it is not word-by-word tanslation. So why do we need to shift the output sequence by one step?
1
vote
1answer
65 views

How are certain machine learning models able to produce variable-length outputs given variable-length inputs?

Most machine learning models, such as multilayer perceptrons, require a fixed-length input and output, but generative (pre-trained) transformers can produce sentences or full articles of variable ...
1
vote
1answer
54 views

How is the transformers' output matrix size arrived at?

In this tensorflow article, the comments in the code say that MHA should output with one of the dimensions being the sequence length of the query/key. However, that means that the second MHA in the ...
0
votes
1answer
31 views

Sentiment analysis does not handle neturals [closed]

I'm writing some financial tools, I've found highly performant models for question and answering but when it comes to sentiment analysis I haven't found anything that good. I'm trying to use ...
1
vote
0answers
28 views

What does the outputlayer of BERT for masked language modelling look like?

In the tutorial BERT – State of the Art Language Model for NLP the masked language modeling pre-training steps are described as follows: In technical terms, the prediction of the output words ...
2
votes
3answers
118 views

What kind of word embedding is used in the original transformer?

I am currently trying to understand transformers. To start, I read Attention Is All You Need and also this tutorial. What makes me wonder is the word embedding used in the model. Is word2vec or GloVe ...
0
votes
0answers
29 views

What is the purpose of “alignment” in the self-attention mechanism of transformers?

I've been reading about transformers & have been having some difficulty understanding the concept of alignment. Based on this article Alignment means matching segments of original text with their ...
1
vote
0answers
31 views

How does the embeddings work in vision transformer from paper?

I get the part from the paper where the image is split into P say 16x16 (smaller images) patches and then you have to ...
0
votes
0answers
78 views

Is using a LSTM, CNN or any other neural network model on top of a Transformer(using hidden states) overkill?

I have recently come across transformers, I am new to Deep Learning. I have seen a paper using CNN and BiLSTM on top of a transformer, the paper uses a transformer(XLM-R) for sentiment analysis in ...
0
votes
0answers
25 views

Is the dimensionality of the query and key tensors in this implementation of the transformer's self-attention correct?

Looking at this tutorial matmul_qk = tf.matmul(q, k, transpose_b=True) # (..., seq_len_q, seq_len_k) is the output of two tensors multiplied of shape: ...
0
votes
1answer
350 views

Transformers: How to use the target mask properly?

I try to apply Transformers to an unusual use case - predict the next user session based on the previous one. A user session is described by a list of events per second, e.g. whether the user watches ...
1
vote
0answers
78 views

Using Reformer model to do NER in Trax

I have been looking at the NER example with Trax in this notebook: https://github.com/google/trax/blob/master/trax/examples/NER_using_Reformer.ipynb However the notebook only gives an example as far ...
2
votes
2answers
115 views

Recent deep learning textbook (i.e. covering at least GANs, LSTM and transformers and attention)

I am searching for an academic (i.e. with maths formulae) textbook which covers (at least) the following: GAN LSTM and transformers (e.g. seq2seq) Attention mechanism The closest match I got is Deep ...
1
vote
0answers
190 views

What is MNLI-(m/mm)?

I came across the term MNLI-(m/mm) in Table 1 of the paper BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. I know what MNLI stands for, i.e. Multi-Genre Natural ...
0
votes
1answer
47 views

Can the attention mechanism improve the performance in the case of short sequences?

I am aware that the attention mechanism can be used to deal with long sequences, where problems related to gradient vanishing and, more generally, representing effectively the whole sequence arise. ...
1
vote
1answer
38 views

In attention models with multiple layers, are weight matrices shared across layers?

In articles that describe neural architectures with multiple attention layers of the same form, are the weight matrices usually the same across the layers? Consider as an example, "Attention is ...
1
vote
2answers
110 views

In the multi-head attention mechanism of the transformer, why do we need both $W_i^Q$ and ${W_i^K}^T$?

In the Attention is all you need paper, on the 4th page, we have equation 1, which describes the self-attention mechanism of the transformer architecture $$ \text { Attention }(Q, K, V)=\operatorname{...
2
votes
0answers
81 views

How to handle long sequences with transformers?

I have a time series sequence with 10 million steps. In step $t$, I have a 400 dimensional feature vector $X_t$ and a scalar value $y_t$ which I want to predict during inference time and I know during ...
3
votes
2answers
192 views

What is different in each head of a multi-head attention mechanism?

I have a difficult time understanding the "multi-head" notion in the original transformer paper. What makes the learning in each head unique? Why doesn't the neural network learn the same ...