Questions tagged [encoder-decoder]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers

Multi-task learning using single encoder + single decoder like structure?

It seems that a lot of researchers predominantly use single encoder + multiple decoders like structure to achieve multi-task learning in computer vision. Would it be reasonable to achieve the multi-...
0 votes
0 answers

Left-to-Right vs Encoder-decoder Models

Xu et al. (2022) distinguishes between popular pre-training methods for language modeling: (see Section 2.1 PRETRAINING METHODS) Left-to-Right: Auto-regressive, Left-to-right models, predict the ...
1 vote
0 answers

How to Train a Decoder for Pre-trained BERT Transformer-Encoder?

Context: I am currently working on an encoder-decoder sequence to sequence model that uses a sequence of word embeddings as input and output, and then reduces the dimensionality of the word embeddings....
0 votes
0 answers

Understanding Probabilty in NEURAL MACHINE TRANSLATION

I am reading the paper "Neural Machine Translation by Jointly Learning to Align and Translate" (PDF), (May 19, 2016), by Bahdanau, Cho and Bengio. I am having trouble with equation 2, page 3:...
0 votes
1 answer

What is a "mask" in the context o RNN-based encoders?

While reading source code related to RNN encoders, I've come across the term mask as input to the encoder. What exactly is it?
  • 3,371
2 votes
1 answer

What is input (and shape) to K/V/Q of self-attention of EACH Decoder block of Language-translation model Transformer's tokens during Inference?

Transformer model of the original Attention paper has a decoder unit that works differently during Inference than Tranining. I'm trying to understand the shapes used during decoder (both self-...
  • 171
0 votes
1 answer

Seq2Seq model produces repeating words

My framework is an encoder-decoder (LSTM-to-LSTM) model, similar to this post. The model basically reads a sentence and generate another sentence. But, the thing is, after a few epochs training, the ...
1 vote
1 answer

How is the transformers' output matrix size arrived at?

In this tensorflow article, the comments in the code say that MHA should output with one of the dimensions being the sequence length of the query/key. However, that means that the second MHA in the ...
  • 111
10 votes
4 answers

What exactly is a hidden state in an LSTM and RNN?

I'm working on a project, where we use an encoder-decoder architecture. We decided to use an LSTM for both the encoder and decoder due to its hidden states. In my specific case, the hidden state of ...
3 votes
1 answer

Why do we need both encoder and decoder in sequence to sequence prediction?

Why do we need both encoder and decoder in sequence to sequence prediction? We could just have a single RNN that, given input $x$, outputs some value $y(t)$ and hidden state $h(t)$. Next, given $h(t)$...