Skip to main content

Questions tagged [seq2seq]

For questions related to sequence-to-sequence (seq2seq) machine learning models/architectures, used e.g. in machine translation.

18 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
3 votes
1 answer
669 views

Is seq2seq the best model when input/output sequences have fixed length?

I understand that seq2seq models are perfectly suitable when the input and/or the output have variable lengths. However, if we know exactly the input/output sequence lengths of the neural network. Is ...
Petrus's user avatar
  • 31
3 votes
0 answers
99 views

What is the difference between zero-padding and character-padding in Recurrent Neural Networks?

For RNN's to work efficiently, we vectorize the operations, which results in an input matrix of shape (m, max_seq_len) where m ...
PhysicsMan's user avatar
2 votes
0 answers
2k views

Any models for text to json

There are many sequence to sequence (seq2seq) models and end to end models, like text to sql. I was wondering are there any text to json deep learning models? For example: Text ...
tired and bored dev's user avatar
2 votes
0 answers
42 views

Are there any successful applications of transformers of small size (<10k weights)?

In the problems of NLP and sequence modeling, the Transformer architectures based on the self-attention mechanism (proposed in Attention Is All You Need) have achieved impressive results and now are ...
spiridon_the_sun_rotator's user avatar
2 votes
0 answers
1k views

What is the time complexity of the forward pass and back-propagation of the sequence-to-sequence model with and without attention?

I keep looking through the literature, but can't seem to find any information regarding the time complexity of the forward pass and back-propagation of the sequence-to-sequence RNN encoder-decoder ...
user1234544's user avatar
1 vote
0 answers
34 views

How to Interpret Cross Attention

I am a bit confused on what cross attention mechanisms are doing. I understand that the currently decoded output is usually the query and the conditioning/input (from an encoder) is the key and value. ...
Kiran Manicka's user avatar
1 vote
0 answers
32 views

Modifying Cross Entropy Loss to work with multiple correct target sequences?

Let's say I'm training a transformer model to perform a seq to seq task, but there are multiple correct answers. For example, the following outputs would all be considered correct: source: A B C -> ...
Brayden Alexander Rudisill's user avatar
1 vote
0 answers
45 views

The model's accuracy becomes suddenly so unreasonably good at beginning of the training process. I need an explaination

I am practicing machine translation using seq2seq model (more specifically with GRU/LSTM units). The following is my first model: This model first archived about 0.03 accuracy score and gradually ...
Đạt Trần's user avatar
1 vote
1 answer
532 views

How to Train a Decoder for Pre-trained BERT Transformer-Encoder?

Context: I am currently working on an encoder-decoder sequence to sequence model that uses a sequence of word embeddings as input and output, and then reduces the dimensionality of the word embeddings....
node_env's user avatar
1 vote
0 answers
96 views

When training a seq2seq model is it better to train using the models outputs or expected outputs?

When training any seq2seq model you have a target and a source. The source may be a sentence ...
Recessive's user avatar
  • 1,396
1 vote
0 answers
176 views

What is the best way to generate German paraphrases?

What is the best method to generate German paraphrases? The state-of-the-art are seq2seq transformer models, like T5, but they only work for English sentences. I found the multilingual MT5 model, but ...
lattenjoe's user avatar
1 vote
0 answers
86 views

Training seq2seq translation model with one source and multiple target

So basically I'm training a sequence to sequence model that translates English sentences to Arabic sentences. I'm using the data provided by Anki @ manythings. I realized that some of the sentences in ...
OmarAI's user avatar
  • 11
1 vote
0 answers
144 views

SeqGAN - Policy gradient objective function interpretation

Could someone clear my doubt on the loss function used in SeqGAN paper . The paper uses policy gradient method to train the generator which is a recurrent neural network here. Have I interpreted the ...
Mathav Raj's user avatar
1 vote
1 answer
145 views

Why feeding the correct output as input during training of seq2seq models?

I've read about seq2seq for time-series and it seemed really promising, but, when I went to implement it, all the tutorials I've found use the correct output as input to the decoder phase during ...
Ramon Dettmer's user avatar
0 votes
1 answer
20 views

probability intepreter of attention mechanism in Seq2Seq

Many people explained seq2seq model by explanatory description. However, in my opinion, that is just like a robot who could say something correctly but don't really understand it. Just like the AI did....
tangyao's user avatar
0 votes
0 answers
8 views

An issue about the Decoder in seq2seq(rnn)

Here is a confusion about the decoder in seq2seq. In each time-step in decoder, there are two outputs: 1.output 2.hidden. and this hidden state is used as the next input hidden state. this output is ...
tangyao's user avatar
0 votes
0 answers
81 views

Seq2seq with attention model only predicts index 0 in a text generation task

I am trying to build a model which summarizes some brief input text (X) and generates a headline (Y). The problem is that the model always outputs 0 as the vocabulary index, which I think that might ...
javi11br's user avatar
0 votes
1 answer
445 views

Fine Tuning Transformer Model for Machine Translation

I am working on the Transformer example demonstrated on TensorFlow's website. https://www.tensorflow.org/text/tutorials/transformer In this example, Machine Translation model is trained to translate ...
boyaronur's user avatar
  • 101