Questions tagged [transformer]

For questions related to the transformer, which is a deep machine learning model introduced in 2017 in the paper "Attention Is All You Need", used primarily in the field of natural language processing (NLP).

Filter by
Sorted by
Tagged with
1
vote
1answer
22 views

What is the Intermediate (dense) layer in between attention-output and encoder-output dense layers within transformer block in PyTorch implementation?

In PyTorch, transformer (BERT) models have an intermediate dense layer in between attention and output layers whereas the BERT and Transformer papers just mention the attention connected directly to ...
0
votes
0answers
11 views

Is there any point in adding the position embedding to the class token in Transformers?

The popular implementations of ViTs by Ross Wightman and Phil Wang add the position embedding to the class tokens as well as to the patches. Is there any point in doing so? The purpose of introduction ...
1
vote
1answer
30 views

What are the major layers in a Vision Transformer?

Currently, I am studying deepfake detection using deep learning methods. Convolution neural networks, recurrent neural networks, long-short term memory networks, and vision transformers are famous ...
1
vote
0answers
25 views

Why does research on faster Transformers focus on the query-key product?

A lot of recent research on Transformers has been devoted to reducing the cost of the self-attention mechanism: $$\text{softmax}\left(\frac{Q K^T}{\sqrt{d}} \right)V,$$ As I understand it, the runtime,...
2
votes
2answers
88 views

In layman terms, what does "attention" do in a transformer?

I heard from many people about the paper titled Attention Is All You Need by Ashish Vaswani et al. What actually does the "attention" do in simple terms? Is it a function, property, or some ...
1
vote
1answer
23 views

Can I use the transformers for the prediction of historical data?

Can I use the transformers for the prediction of wind power with the historical data? Dataset Datetime, Ambient temperature (Degree), Dewpoint (Degree), Relative Humidity\n (%), Air Pressure, Wind ...
1
vote
0answers
20 views

What is the purpose of hard distillation?

In order to get a smaller model, one often uses larger model, that performs reasonably well on the data as a teacher, and uses the information from large model to train the smaller one. There are ...
0
votes
0answers
23 views

Is the multi-head attention in the transformer a weighted adjacency matrix?

Are multi-head attention matrices weighted adjacency matrices? The job of the multi-head-attention mechanism in transformer models is to determine how likely a word is to appear after another word. In ...
0
votes
0answers
11 views

Positional Encoding for adjacency matrix

I understand that a normal positional encoding helps a transformer to understand pictures better and that it allows the (otherwise permutational invariant transformer-network) to create relationships ...
-1
votes
1answer
36 views

Can I use Sentence-Bert for embedding for fake news detection?

What are the benefits of Sentence-Bert for Sentence Embedding vs other Embedding models like Bert?
0
votes
0answers
11 views

What is the difference between supervised and unsupervised training in T5?

I know unsupervised training for T5 is like: input: He went X output: X to school Z is this equivalent to the following in a supervised manner: ...
2
votes
1answer
70 views

How to generate a response while considering past questions as well?

User: What is the tallest mountain? Agent: Everest User: Where is it located? # Agent hears: "Where is Everest located?" Agent: Nepal I want to be able ...
1
vote
2answers
34 views

Why do the authors of the T5 paper say that the "architectural changes are orthogonal to the experimental factors"?

Here's a quote from the T5 paper (T5 stands for "Text-to-Text Transfer Transformer") titled Exploring the Limits of Transfer Learning with a Unified Text-...
0
votes
0answers
39 views

How does bipartite matching work in DETR? [closed]

I was going through the DETR paper to understand this end-to-end detection transformer used for object detection, and I came across this bipartite matching thing.
0
votes
0answers
11 views

One single-batch training on Huggingface Bert model "ruins" the model

For some reason, I need to do further (2nd-stage) pre-training on Huggingface Bert model, and I find my training outcome is very bad. After debugging for hours, surprisingly, I find even training one ...
0
votes
0answers
20 views

Proper initialization of the weight matrices in MultiheadAttention

Self-attention layers have 4 learnable tensors (in the vanilla formulation): Query matrix $W_Q$ Key matrix $W_K$ Value matrix $W_V$ Output matrix $W_O$ Nice illustration from https://jalammar....
0
votes
1answer
76 views

Transformer model is very slow and doesn't predict well

I created my first transformer model, after having worked so far with LSTMs. I created it for multivariate time series predictions - I have 10 different meteorological features (temperature, humidity, ...
2
votes
1answer
81 views

Why does a transformer not use an activation function following the multi-head attention layer?

I was hoping someone could explain to me why in the transformer model from the "Attention is all you need" paper there is no activation applied after both the multihead attention layer and ...
1
vote
1answer
28 views

How is Google Translate able to translate texts of arbitrarily large length?

Sequence-to-sequence models with attention are known to be limited by a maximum sequence length. So how can we handle sequences of arbitrarily large size? Do we just set a very large maximum sequence ...
0
votes
0answers
19 views

Which Neural Network Topology to choose, are Transformers suitable?

I have a regression problem and I am not quite sure which architecture to choose. I never worked with transformers before, but I generally understand how they work and I think they might be suitable. ...
0
votes
1answer
86 views

What is the difference between a vision transformer and image based relational learning

I am trying to figure out the difference between the architecture used in this and this paper. It looks like both used multi-headed self-attention and therefore should be the same in principle.
0
votes
1answer
16 views

What is the bit memory task?

I learned from this post about the so-called bit memory: They froze its self-attention and feed-forward layers and, in separate copies, fine-tuned peripheral layers on each on a wide range of tasks: ...
0
votes
1answer
38 views

Why people always say the Transformer is parallelizable while the self-attention layer still depends on outputs of all time steps to calculate?

When compared to an RNN seq-to-seq model, people always say the Transformer is parallelizable. In the original Attention Is All You Need paper, it also said that Recurrent models typically factor ...
1
vote
0answers
25 views

Positional Encoding in Transformer on multi-variate time series data hurts performance

I set up a transformer model that embeds positional encodings in the encoder. The data is multi-variate time series-based data. As I just experiment with the positional encoding portion of the code I ...
3
votes
1answer
82 views

In Computer Vision, what is the difference between a transformer and attention?

Having been studying computer vision for a while, I still cannot understand what the difference between a transformer and attention is?
0
votes
0answers
16 views

Computing the mean attention distance for ViT

Recently I came across the paper that introduces the Vision Transformer (ViT) "AN IMAGE IS WORTH 16X16 WORDS: TRANSFORMERS FOR IMAGE RECOGNITION AT SCALE". The thing I don't really ...
1
vote
1answer
63 views

Continuous sequence data with Transformer model

What is the right way to input continuous, temporal(time series) data into Transformer network. Assume we're using the basic TransformerBlock here. Since data is continuous with no tokens, Token ...
0
votes
0answers
22 views

Keras MLP performing better than Transformers

I'm working on a comparative study using some models in a sentiment analysis task: MLPs and LSTMs with and without the use of word embeddings (GloVe and Word2Vec) and two Transformer-based models (...
0
votes
0answers
7 views

Can I use transfer learning with Bert and none text sequential data?

I'm working on a multiclass classification problem, each row on my dataset have 5 time windows with 23 values on each time window, I would like to use transfer learning using the Bert transformer to ...
1
vote
0answers
24 views

Why class embedding token is added to the Visual Transformer?

In the famous work on the Visual Transformers, the image is split into patches of a certain size (say 16x16), and these patches are treated as tokens in the NLP tasks. In order to perform ...
0
votes
0answers
32 views

Why and when transformers are better than CNN's in sequence modeling tasks?

Transformers have made a revolution in the domain of NLP and gave rise to a rapid boost of neural networks in a variety of language modelling problems, TTS and, recently, achieved competitive accuracy ...
0
votes
0answers
53 views

What is the best way to generate German paraphrases?

What is the best method to generate German paraphrases? The state-of-the-art are seq2seq transformer models, like T5, but they only work for English sentences. I found the multilingual MT5 model, but ...
0
votes
0answers
14 views

Does the transformers model (in "Attention is All You Need") exclude the encoder in language modelling tasks?

The language model I am referring to is the one outlined in "Attention is All You Need": My understanding is that when the task is translation, the encoder's input could be "Hi, my ...
0
votes
0answers
13 views

MBART and better Domain Specific Translations Using Masks?

I'm implementing https://huggingface.co/facebook/mbart-large-50-many-to-many-mmt for translations as it has shown promising results but I wanted to see if there was a way to translate specific parts ...
0
votes
0answers
51 views

How do autoregressive attention mechanism work in multi-headed attention?

[LONG POST!!] I am working on a DNN model that works as an improviser to generate music sequences. The idea of generating music is based on taking a sequence of music nodes (their index representation)...
0
votes
0answers
26 views

Positional encoding in convolutional layers

Positional encoding (PE) is an essential part of the self-attention layers in the transformer architectures since without adding it in some way (fixed of learnable) to the input embeddings model has ...
0
votes
0answers
12 views

Visualizing encoder-attention after ResNet in terms of ResNet input

I have a transform-encoder only architecture, which has the following structure: ...
2
votes
1answer
39 views

What makes a transformer a transformer?

Transformers are modified heavily in recent research. But what exactly makes a transformer a transformer? What is the core part of a transformer? Is it the self-attention or the parallelism or ...
2
votes
2answers
97 views

Is it realistic to train a transformer-based model (e.g. GPT) in a self-supervised way directly on the Mel spectrogram?

In music information retrieval, one usually converts an audio signal into some kind "sequence of frequency-vectors", such as STFT or Mel-spectrogram. I'm wondering if it is a good idea to ...
0
votes
0answers
10 views

How does translation of a text in an HTML tag works? Consider the following cases

Recently I came a cross to the Google Translate's add-on for Chrome. Basically you have a web page and it translates that web page for you while keeping its structure. I came to the conclusion that it ...
1
vote
0answers
21 views

Are there any successful applications of transformers of small size (<10k weights)?

In the problems of NLP and sequence modeling Transformer architectures based on self-attention mechanism https://arxiv.org/abs/1706.03762 have achieved impressive results and now are the first choices ...
0
votes
0answers
53 views

Is it sensible to combine GRU/LSTM with the transformer's encoder?

Is it sensible to combine GRU/LSTM with the transformer's encoder? If we take the output of a GRU (uni or bi-directional), and then feed it as input to the transformer's encoder, would that help in ...
1
vote
1answer
59 views

How do transformers understand data and answer custom questions?

I recently heard of GPT-3 and I don't understand how the attention models and transformers encoders and decoders work. I heard that GPT-3 can make a website from a description and write perfectly ...
0
votes
0answers
27 views

Representing variable-length sequences

I want to train a model over a variable-length sequential data (e.g. the temperature at different times of day) where the output depends on what the temperature is at time ...
2
votes
0answers
15 views

Embedding from Transformer-based model from paragraph or documnet (like Doc2Vec)

I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this: ...
0
votes
0answers
16 views

Can transformers be used to improve regression?

I was recently reading a bit about transformers and I don't understand them very much but I was wondering if anyone knows if any of their techniques such as attention mechanism or anything has been ...
2
votes
0answers
21 views

What part of the Vaswani et al. is the "transformer"?

Which part of this is the transformer? Ok, the caption says the whole thing is the transformer, but that's back in 2017 when the paper was published. My question is about how the community uses the ...
2
votes
0answers
20 views

How to Select Model Parameters for Transformer (Heads, number of layers, etc)

Is there a general guideline on how the Transformer model parameters should be selected, or the range of these parameters that should be included in a hyperparameter sweep? Number of heads Number of ...
0
votes
1answer
46 views

Can an existing transformer model be modified to estimate the next most probable number in a sequence of numbers?

Models based on the transformer architectures (GPT, BERT, etc.) work awesome for NLP tasks including taking an input generated from words and producing probability estimates of the next word as the ...
3
votes
2answers
230 views

Why does GPT-2 Exclude the Transformer Encoder?

After looking into transformers, BERT, and GPT-2, from what I understand, GPT-2 essentially uses only the decoder part of the original transformer architecture and uses masked self-attention that can ...