Questions tagged [encoder-decoder]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
1 answer
44 views

What is a "mask" in the context o RNN-based encoders?

While reading source code related to RNN encoders, I've come across the term mask as input to the encoder. What exactly is it?
user avatar
  • 3,099
0 votes
0 answers
16 views

U-Net with freezed encoder and customize decoder

I want to use a U-Net architecture, but I want only to use the feature extraction part and train the decoder for my task, but I'm not sure if the decoder of U-Net would be enough for learning new ...
user avatar
  • 33
0 votes
1 answer
315 views

What is input (and shape) to K/V/Q of self-attention of EACH Decoder block of Language-translation model Transformer's tokens during Inference?

Transformer model of the original Attention paper has a decoder unit that works differently during Inference than Tranining. I'm trying to understand the shapes used during decoder (both self-...
user avatar
  • 151
0 votes
0 answers
66 views

What would be the total number of learnable parameters of the RNN encoder of this encoder-decoder architecture for machine translation?

Here's a quiz. My answer is different from the teacher's, so I'm wondering what answer would you pick up. We use a sequence-to-sequence (encoder-decoder) system to perform machine translation. We ...
user avatar
  • 113
0 votes
1 answer
313 views

Seq2Seq model produces repeating words

My framework is an encoder-decoder (LSTM-to-LSTM) model, similar to this post. The model basically reads a sentence and generate another sentence. But, the thing is, after a few epochs training, the ...
user avatar
1 vote
1 answer
88 views

How is the transformers' output matrix size arrived at?

In this tensorflow article, the comments in the code say that MHA should output with one of the dimensions being the sequence length of the query/key. However, that means that the second MHA in the ...
user avatar
  • 111
8 votes
4 answers
11k views

What exactly is a hidden state in an LSTM and RNN?

I'm working on a project, where we use an encoder-decoder architecture. We decided to use an LSTM for both the encoder and decoder due to its hidden states. In my specific case, the hidden state of ...
user avatar
3 votes
1 answer
1k views

Why do we need both encoder and decoder in sequence to sequence prediction?

Why do we need both encoder and decoder in sequence to sequence prediction? We could just have a single RNN that, given input $x$, outputs some value $y(t)$ and hidden state $h(t)$. Next, given $h(t)$...
user avatar