Questions tagged [encoder-decoder]
The encoder-decoder tag has no usage guidance.
10
questions
0
votes
0
answers
9
views
Multi-task learning using single encoder + single decoder like structure?
It seems that a lot of researchers predominantly use single encoder + multiple decoders like structure to achieve multi-task learning in computer vision. Would it be reasonable to achieve the multi-...
0
votes
0
answers
20
views
Left-to-Right vs Encoder-decoder Models
Xu et al. (2022) distinguishes between popular pre-training methods for language modeling: (see Section 2.1 PRETRAINING METHODS)
Left-to-Right:
Auto-regressive, Left-to-right models, predict the ...
- 29
1
vote
0
answers
102
views
How to Train a Decoder for Pre-trained BERT Transformer-Encoder?
Context:
I am currently working on an encoder-decoder sequence to sequence model that uses a sequence of word embeddings as input and output, and then reduces the dimensionality of the word embeddings....
- 11
0
votes
0
answers
21
views
Understanding Probabilty in NEURAL MACHINE TRANSLATION
I am reading the paper "Neural Machine Translation by Jointly Learning to Align and Translate" (PDF), (May 19, 2016), by Bahdanau, Cho and Bengio.
I am having trouble with equation 2, page 3:...
0
votes
1
answer
84
views
What is a "mask" in the context o RNN-based encoders?
While reading source code related to RNN encoders, I've come across the term mask as input to the encoder. What exactly is it?
- 3,371
2
votes
1
answer
853
views
What is input (and shape) to K/V/Q of self-attention of EACH Decoder block of Language-translation model Transformer's tokens during Inference?
Transformer model of the original Attention paper has a decoder unit that works differently during Inference than Tranining.
I'm trying to understand the shapes used during decoder (both self-...
- 171
0
votes
1
answer
464
views
Seq2Seq model produces repeating words
My framework is an encoder-decoder (LSTM-to-LSTM) model, similar to this post. The model basically reads a sentence and generate another sentence. But, the thing is, after a few epochs training, the ...
1
vote
1
answer
126
views
How is the transformers' output matrix size arrived at?
In this tensorflow article, the comments in the code say that MHA should output with one of the dimensions being the sequence length of the query/key. However, that means that the second MHA in the ...
- 111
10
votes
4
answers
15k
views
What exactly is a hidden state in an LSTM and RNN?
I'm working on a project, where we use an encoder-decoder architecture. We decided to use an LSTM for both the encoder and decoder due to its hidden states. In my specific case, the hidden state of ...
- 647
3
votes
1
answer
1k
views
Why do we need both encoder and decoder in sequence to sequence prediction?
Why do we need both encoder and decoder in sequence to sequence prediction?
We could just have a single RNN that, given input $x$, outputs some value $y(t)$ and hidden state $h(t)$. Next, given $h(t)$...
- 51