Questions tagged [transformer]

For questions related to the transformer, which is a deep machine learning model introduced in 2017 in the paper "Attention Is All You Need", used primarily in the field of natural language processing (NLP).

Filter by
Sorted by
Tagged with
0 votes
0 answers
5 views

How to remove boilerplate (or extract main content) from web pages?

Data: Raw source code of a website and the final cleaned main content I want to extract from the raw source code. The source code comes from different websites with different layouts and code ...
0 votes
0 answers
9 views

parsing a text to some set of known parameters

Suppose I have a set of sentences describing reviews: The car is from brand X, drove Y kilometers , and is in great condition A ...
0 votes
1 answer
25 views

Are neural networks a strict special case of a transformer?

Since transformers contain a neural network, are they a strict generalisation of standard feedforward neural networks? In what ways can transformers be interpreted as a generalisation and abstraction ...
0 votes
0 answers
19 views

Do Transformers and LSTMs use the same word embeddings (except for the position encoding, which only Transformers use)?

In NLP, the first step is always to "convert" the given words of a sentence into representation vectors (word embeddings). As I understand it, in the case of transformers, the words/...
  • 1
0 votes
0 answers
11 views

How to get sentence from embedding vector with Universal Sentence Encoder?

Given a sentence embedding vector from a Sentence Encoder (like Sentence-BERT), I want to train a model to generate the original sentence (list of word embedding). Are there any architectures to ...
1 vote
0 answers
37 views

How to Train a Decoder for Pre-trained BERT Transformer-Encoder?

Context: I am currently working on an encoder-decoder sequence to sequence model that uses a sequence of word embeddings as input and output, and then reduces the dimensionality of the word embeddings....
0 votes
0 answers
13 views

Surrogate model to produce time series from parameter set

Say I have a model $M$ that takes in a parameter vector $\beta$, and produces a (numerical) time series. This could be a complicated model (e.g. a bespoke enzyme reaction model), or something simple ...
  • 1
2 votes
0 answers
18 views

Using similarity score within lstm embedding for attention based mechanism

Yesterday, I found this fascinating paper about predicting various clinical conditions using an attention based LSTM. I don't have any practical experience with attention mechanism or transformers, ...
  • 21
0 votes
0 answers
4 views

Do ML Models with sparsely accessed layers in the middle or end exist?

I am currently researching ML training, specifically if layers are accessed in a sparse or dense fashion. A linear layer an example for dense access, as all parameters are required during the forward ...
  • 1
0 votes
0 answers
19 views

Understanding Probabilty in NEURAL MACHINE TRANSLATION

I am reading the paper "Neural Machine Translation by Jointly Learning to Align and Translate" (PDF), (May 19, 2016), by Bahdanau, Cho and Bengio. I am having trouble with equation 2, page 3:...
0 votes
1 answer
32 views

Shuffling vs Non-shuffling train/test set yields drastically different results

I am currently working with a very deep NN (200mio. to 350mio. params). My data set is roughly of shape (2mio, 350), i.e. 2mio samples and 350 features. In fact, the features are time series. As input ...
0 votes
0 answers
7 views

Shuffle data inside learning sample in order independet transformer model

Does it make sense to create new samples with shuffled items "tokens" inside a learning sample for the order independent (no positional encoding) transformer model to improve model accuracy?
0 votes
0 answers
12 views

Transformers (NLP) High Accuracy, Low Loss

What does accuracy and loss mean during the training phrase for seq2seq Transformer Models (I'm using the keras API). I'm getting high accuracy and low loss here but my seq2seq predictions are ...
0 votes
1 answer
102 views

Last linear layer of the decoder of a transformer

I am learning the transformers architecture from these two sources: https://arxiv.org/pdf/1706.03762.pdf https://jalammar.github.io/illustrated-transformer/ I just wanted to ask about the final step ...
  • 101
0 votes
0 answers
14 views

How to embed relational information in a Transformer for NMT task?

I have AMR graph like the following: I am using Transformer model for machine translation. However, my input data has relational information as shown above. This information has semantic information ...
2 votes
0 answers
32 views

What is the most important predecessor of the transformer model?

I'm wondering what the origins of the transformer as proposed in Attention Is All You Need are. The paper itself provides some interesting pointers to the literature on self-attention such as: A ...
0 votes
0 answers
7 views

Difference in mask in the end-to-end transformer model>

In the book Deep Learning with Python, 2nd edition François Chollet writes (section 11.5.3. listing 11.36, page 361): ...
0 votes
0 answers
35 views

Dall-E Question (probably silly)

I have started to read the Dall-E (1) paper and I have a quick question that will help me a lot. I know the basis of Transformers but only for NLP tasks (Text-to-text). So when i read : "The ...
0 votes
0 answers
80 views

why cross entropy loss has to be multiplied by a batch size during an evaluation in transformer model?

I am trying to look through a code of the transformer model from Pytorch. However, I do not understand why batch size needs to multiply with cross-entropy loss given that loss is calculated based on ...
  • 1
0 votes
0 answers
25 views

Is there benefit to autoregressive models for deep RL tasks with long episodes and short required context?

General Case In deep RL (specifically in the space of policy gradient methods) it seems very common that encoder-decoder models (either transformer or RNN-variant) are used in the policy/value ...
  • 101
1 vote
1 answer
48 views

Rationalle behind SE3 Transformer?

I have just finished reading the SE3 transformer paper (SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks) by Fuchs et-al and although I'm sure I understand less than 50% of the ...
0 votes
1 answer
130 views

How special tokens in BERT-Transformers work?

"[SEP] tokens are useful to differentiate the questions from answers through type_ids" Yes, but how is this helping model to understand that "I should look paragraph and generate ...
  • 1
0 votes
1 answer
79 views

How does GPT use the same embedding matrix for both input and output?

My understanding is that GPT uses the same embedding matrix for both inputs and output: Let $V$ be the vocab size, $D$ the number of embedding dimensions, and $E$ be a $V \times D$ embedding matrix: ...
0 votes
0 answers
20 views

How should I classify the genre of a youtube channel

It turns out that classifying a channel's genre isn't a simple task, as in most cases it is not specified by the content creator. To do such a task, I thought of the following pipeline: Preprocess: ...
3 votes
1 answer
96 views

Why are embeddings added, not concatenated?

Let's consider the following example from BERT I cannot understand why "the input embeddings are the sum of the token embeddings, the segmentation embeddings, and the position embeddings". ...
  • 161
0 votes
1 answer
97 views

Why do transformer Key Query Value layers not have biases or activations?

Transformers use just matrices to transform input embeddings, which is halfway to being a connected dense layer (add a bias and activation). So, why don't transformers have dense layers for encoding ...
0 votes
0 answers
15 views

How do we call a transformer having N encoders and M decoders and a learnable cross-connectivity between encoders and decoders?

How do we call a transformer having N encoders and M decoders and a learnable cross-connectivity between encoders and decoders? I am interested particularly in the case when M=1, but I imagine that it ...
  • 51
0 votes
0 answers
12 views

Transformer-XL query length differs during inference and optimization?

I'm working through KakaoBrain's Transformer Reinforcement Learning implementation. https://github.com/kakaobrain/brain_agent I observed that the query length during sampling is set to ...
1 vote
0 answers
36 views

What considerations should I take to train my transformer model?

I want to train my vision transformer model on a benchmark for an image segmentation task: (LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation) (GitHub), but I don't ...
0 votes
0 answers
43 views

Which index of the output in Transformers is used during inference to predict multiple words?

I am somewhat confused about how transformers, not just the original model, but also models like GPT-2 work when they are not training but are used multiple times to predict single tokens/words. The ...
2 votes
1 answer
520 views

When exactly does the split into different heads in Multi-Head-Attention occur?

I am confused by the Multi-Head part of the Multi-Head-Attention used in Transformers. My question concerns the implementations in Pytorch of nn.MultiheadAttention and its forward method ...
2 votes
1 answer
493 views

Positional Encoding of Time-Series features

I’m trying to use a Transformer Encoder I coded with weather feature vectors which are basically 11 features about the weather in the dimension ...
0 votes
0 answers
64 views

Why are separate, bigger Encoder-Decoder architectures used instead of Bidirectional RNNs/Transformers for Seq2Seq tasks?

Whether with RNNs or Transformers, Encoder-Decoder networks are used for Sequence to Sequence (Seq2Seq) tasks, like Machine Translation. Why are separate, bigger Encoder-Decoder networks used for this ...
1 vote
0 answers
39 views

What does "position" in "each position in the decoder" denote in the Transformer's original paper?

I am reading Attention is All You Need and I feel confused about the word "position" in this paper, by the way I'm not native English speaker which may cause my confusion which has confused ...
  • 11
1 vote
0 answers
30 views

When training a seq2seq model is it better to train using the models outputs or expected outputs?

When training any seq2seq model you have a target and a source. The source may be a sentence ...
  • 1,266
0 votes
1 answer
41 views

Attention: Isn't it redundant to apply a linear layer to both the keys and values?

Transformer attention is calculated $Attention(X) =X W^V\times \text{columnwise-softmax} (Att(X))$ where the attention attention matrix is $$Att(X) = Q \times K = {X W}^Q \times ({X W}^K)^T = {X W}^Q (...
0 votes
0 answers
51 views

Does the transformer model have any inherent ordering?

I'm aware of the practice of positional encoding, which inserts information to convey the relative positions of input data points. Unfortunately, I do not have a great grasp of the transformer model ...
0 votes
0 answers
18 views

Why is it common to use K=V in Attention layers?

Context: Hi I recently read from the keras docs: "key: Optional key Tensor of shape (B, S, dim). If not given, will use value for both key and value, which is the most common case." I found ...
  • 296
1 vote
1 answer
81 views

How to use structural information in a Transformer?

I am performing a Neural Machine Translation (NMT) task. In my case, input data has relational information. I know I can use a Graph Neural Network (GNN) and use a Graph2Seq model. But I can't find a ...
0 votes
0 answers
60 views

How to include additional numeric input in the Transformer architecture

I want to apply the Transformer architecture to my machine translation task, and provide the decoder with an additional parameter in the range of [0,1]. This ...
0 votes
0 answers
21 views

Transformers for regression on permutation of fixed size sequence?

Transformers have shown remarkable performance operating on sequences, but are equivariant to the order in the input sequence. Positional Encoding alleviates that problem, but how good is it? In my ...
  • 246
0 votes
0 answers
22 views

CTC Loss incredibly low for wrong output

I am trying to train an OCR model with Vision Transformers. While training the output is a vector with values full of zeros which is obviously padding value. But the CTC loss was small that it was ...
0 votes
0 answers
17 views

What is the canocial way to handling differing input and output dimensions for the transformer model?

I have an essential regression task, where the input is of dimension $d$ and the output is a scalar. I think the transformer model is a good fit for this problem. In the vanilla multi-head-attention ...
  • 51
0 votes
0 answers
43 views

How to interpret Transformer output

In this (https://towardsdatascience.com/a-detailed-guide-to-pytorchs-nn-transformer-module-c80afbc9ffb1) article the author says, that the output of the ...
0 votes
1 answer
175 views

Dimensions of a Transformer model and purpose of masking [closed]

I'm currently studying the Transformer model (Attention is all you need) and after reading it I still have some questions for which I get conflicting answers if I google them: What exactly are the ...
0 votes
1 answer
470 views

How "Patch Merging" works in SWIN-Transformers?

In the SOTA paper: SWIN-Transformers, the authors have tried their best to explain everything clearly. I have got an idea of how it works except the Patch Merging ...
  • 233
0 votes
0 answers
371 views

How do we reduce the output dimensions of BERT?

The output dimensions of BERT are 768-dimensional, is it possible to reduce them to a lower, custom number? For example, if there is another BERT-based transformer model which has a lower count of ...
0 votes
1 answer
25 views

Are positional embeddings computed during or before training?

I'm trying to practically frame the concept of positional embeddings as introduced in the original paper. As far as I've understood, what we do is basically creating some other vectors in addition to ...
2 votes
1 answer
267 views

What is multi-head attention doing mathematically, and how is it different from self-attention?

I'm trying to understand the difference between the concept of self-attention and multi-head attention. The latter is not actually too clear to me. I understand that, in the case of self-attention, we ...
0 votes
1 answer
207 views

Is a Transformer a good choice for multivariate signal classification?

I am working on a problem regarding the multi-classification of multivariate time signals. So I have multiple signals and try to train an algorithm on them. My current approach is to build a neural ...