Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [seq2seq]

For questions related to sequence-to-sequence (seq2seq) machine learning models/architectures, used e.g. in machine translation.

Filter by
Sorted by
Tagged with
1 vote
0 answers
22 views

How to Interpret Cross Attention

I am a bit confused on what cross attention mechanisms are doing. I understand that the currently decoded output is usually the query and the conditioning/input (from an encoder) is the key and value. ...
Kiran Manicka's user avatar
4 votes
2 answers
2k views

What are the differences between seq2seq and encoder-decoder architectures?

I've read many tutorials online that use both words interchangeably. When I search and find that they are the same, why not just use one word since they have the same definition?
user avatar
1 vote
0 answers
28 views

Modifying Cross Entropy Loss to work with multiple correct target sequences?

Let's say I'm training a transformer model to perform a seq to seq task, but there are multiple correct answers. For example, the following outputs would all be considered correct: source: A B C -> ...
Brayden Alexander Rudisill's user avatar
0 votes
1 answer
55 views

How to train a seq2seq model to rephrase input text following given rules

I want to train (fine-tune) a seq2seq model to perform the task of rephrasing input following these rules : 1- always follow the pattern "Entity Verb Entity" 2- only use simple sentences : ...
Wissem Boujlida's user avatar
0 votes
1 answer
56 views

How does GPT like Decoder only conversational models distunguish the source of text?

In a conversational setting where two sources of text (user and the model) follow each other like below User: some text bla bla Model: another text bah bah User: bla bla bla Model: bah bah and so on, ...
meliksahturker's user avatar
0 votes
1 answer
27 views

Seq2Seq model- Confusing about the dimension of Seq2Seq model [closed]

I am new to Seq2Seq and hope to find a proper guildances, advices. I am doing a Project from an online course so I can not give the material but I got my Project notebook on Github I want to ask ...
QH.Chu's user avatar
  • 3
0 votes
0 answers
10 views

What would be the best approach to resume longer texts in one word?

I am trying to create a model capable of resuming longer texts (my dataset has up to 140 words for each instance) in a single word (or multiple separate words). The idea is to synthetize positive or ...
Victor Ferreira's user avatar
1 vote
0 answers
45 views

The model's accuracy becomes suddenly so unreasonably good at beginning of the training process. I need an explaination

I am practicing machine translation using seq2seq model (more specifically with GRU/LSTM units). The following is my first model: This model first archived about 0.03 accuracy score and gradually ...
Đạt Trần's user avatar
0 votes
1 answer
77 views

Seq2seq with RNNs, how is the training loop performed?

How do we train a seq2seq rnn training? We input a sentence that needs to be translated. We encode it sequentially. Then the first decoder outputs the first word with probabilities. We do a gradient ...
FluidMechanics Potential Flows's user avatar
1 vote
1 answer
59 views

Why is it called a Seq2Seq model if the output is just a number?

Why is it called a Seq2Seq model if the output is just a number? For example, if you are trying to predict a movie's recommendation, and you are inputting a ...
Katsu's user avatar
  • 151
0 votes
0 answers
79 views

Seq2seq with attention model only predicts index 0 in a text generation task

I am trying to build a model which summarizes some brief input text (X) and generates a headline (Y). The problem is that the model always outputs 0 as the vocabulary index, which I think that might ...
javi11br's user avatar
0 votes
1 answer
436 views

Fine Tuning Transformer Model for Machine Translation

I am working on the Transformer example demonstrated on TensorFlow's website. https://www.tensorflow.org/text/tutorials/transformer In this example, Machine Translation model is trained to translate ...
boyaronur's user avatar
  • 101
1 vote
1 answer
249 views

Is the decoder in a transformer Seq2Seq model non parallelizable?

From my understanding, seq2seq models work by first computing a representation of the input sequence, and feeding this to the decoder. The decoder then predicts each token in the output sequence in an ...
Andrew Tang's user avatar
2 votes
0 answers
2k views

Any models for text to json

There are many sequence to sequence (seq2seq) models and end to end models, like text to sql. I was wondering are there any text to json deep learning models? For example: Text ...
tired and bored dev's user avatar
1 vote
2 answers
122 views

How does Seq2Seq with attention actually use the attention (i.e. the context vector)?

For neural machine translation, there's this model "Seq2Seq with attention", also known as the "Bahdanau architecture" (a good image can be found on this page), where instead of ...
Mew's user avatar
  • 181
1 vote
1 answer
488 views

How to Train a Decoder for Pre-trained BERT Transformer-Encoder?

Context: I am currently working on an encoder-decoder sequence to sequence model that uses a sequence of word embeddings as input and output, and then reduces the dimensionality of the word embeddings....
node_env's user avatar
1 vote
0 answers
95 views

When training a seq2seq model is it better to train using the models outputs or expected outputs?

When training any seq2seq model you have a target and a source. The source may be a sentence ...
Recessive's user avatar
  • 1,396
1 vote
1 answer
80 views

Does Seq2Seq decoder take a special vector or the weights of the last encoder cell as an output?

I'm reading Sequence to Sequence Learning with Neural Networks and there's a thing that I couldn't quite grasp. Paper says the encoder outputs a vector to be fed to the decoder. More precisely Our ...
J.Smith's user avatar
  • 145
3 votes
1 answer
662 views

Is seq2seq the best model when input/output sequences have fixed length?

I understand that seq2seq models are perfectly suitable when the input and/or the output have variable lengths. However, if we know exactly the input/output sequence lengths of the neural network. Is ...
Petrus's user avatar
  • 31
1 vote
0 answers
176 views

What is the best way to generate German paraphrases?

What is the best method to generate German paraphrases? The state-of-the-art are seq2seq transformer models, like T5, but they only work for English sentences. I found the multilingual MT5 model, but ...
lattenjoe's user avatar
4 votes
1 answer
156 views

Can Reinforcement Learning be used to generate sequences?

Can we use reinforcement learning for sequence-to-sequence tasks? If yes, whether or not this is a good choice, how could this be done?
penguin_smasher's user avatar
2 votes
0 answers
40 views

Are there any successful applications of transformers of small size (<10k weights)?

In the problems of NLP and sequence modeling, the Transformer architectures based on the self-attention mechanism (proposed in Attention Is All You Need) have achieved impressive results and now are ...
spiridon_the_sun_rotator's user avatar
1 vote
0 answers
85 views

Training seq2seq translation model with one source and multiple target

So basically I'm training a sequence to sequence model that translates English sentences to Arabic sentences. I'm using the data provided by Anki @ manythings. I realized that some of the sentences in ...
OmarAI's user avatar
  • 11
0 votes
1 answer
1k views

Seq2Seq model produces repeating words

My framework is an encoder-decoder (LSTM-to-LSTM) model, similar to this post. The model basically reads a sentence and generate another sentence. But, the thing is, after a few epochs training, the ...
Cheleeger Ken's user avatar
3 votes
0 answers
98 views

What is the difference between zero-padding and character-padding in Recurrent Neural Networks?

For RNN's to work efficiently, we vectorize the operations, which results in an input matrix of shape (m, max_seq_len) where m ...
PhysicsMan's user avatar
5 votes
1 answer
2k views

What's the difference between content-based attention and dot-product attention?

I'm following this blog post which enumerates the various types of attention. It mentions content-based attention where the alignment scoring function for the $j$th encoder hidden state with respect ...
Alexander Soare's user avatar
2 votes
1 answer
452 views

How is Google Translate able to convert texts of different lengths?

According to my experience with Tensorflow and many other frameworks, neural networks have to have a fixed shape for any output, but how does Google translate convert texts of different lengths?
Gavin's user avatar
  • 49
1 vote
0 answers
144 views

SeqGAN - Policy gradient objective function interpretation

Could someone clear my doubt on the loss function used in SeqGAN paper . The paper uses policy gradient method to train the generator which is a recurrent neural network here. Have I interpreted the ...
Mathav Raj's user avatar
2 votes
0 answers
1k views

What is the time complexity of the forward pass and back-propagation of the sequence-to-sequence model with and without attention?

I keep looking through the literature, but can't seem to find any information regarding the time complexity of the forward pass and back-propagation of the sequence-to-sequence RNN encoder-decoder ...
user1234544's user avatar
1 vote
1 answer
970 views

Do Seq2Seq models and the Bidirectional RNN do the same thing?

It seems to me that Seq2Seq models and Bidirectional RNNs try to do the same thing. Is that true? Also, when would you recommend one setup over another?
ADSBJason's user avatar
16 votes
4 answers
29k views

What exactly is a hidden state in an LSTM and RNN?

I'm working on a project, where we use an encoder-decoder architecture. We decided to use an LSTM for both the encoder and decoder due to its hidden states. In my specific case, the hidden state of ...
user8714896's user avatar
1 vote
1 answer
144 views

Why feeding the correct output as input during training of seq2seq models?

I've read about seq2seq for time-series and it seemed really promising, but, when I went to implement it, all the tutorials I've found use the correct output as input to the decoder phase during ...
Ramon Dettmer's user avatar