6
$\begingroup$

The basic seq-2-seq model consists of 2 parts: a recurrent encoder that compresses a sequence to a vector and decoder that unrolls the vector into the output sequence:

enter image description here

Why is the output, w, x, y, z of the decoder used as its input? Shouldn't the hidden state of the RNN from the previous timestamps be enough?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
3
$\begingroup$

In seq2seq they model the joint distribution of whatever char/word sequence by decomposing it into time-forward conditionals:

\begin{align*} p(w_1,w_2, \dots,w_n) &= \ p(w_1)*p(w_2|w_1) * \ ... \ * p(w_n|w_1, \dots,w_{n-1}) \\ &= \ p(w_1)*\prod_{i=2}^{n}p(w_i|w_{<i}) \end{align*}

This can be sampled by sampling each of the conditional in ascending order. So, that's exactly what they're trying to imitate. You want the second output dependant on the sampled first output, not its distribution.

This is why the hidden state is NOT good for modeling this setup because it is a latent representation of the distribution, not a sample of the distribution.

Note: In training, they use ground-truth as input by default because it working under the assumption the model should've predicted the correct word, and, if it didn't, the gradient of the word/char level loss will reflect that (this is called teacher forcing and has a multitude of pitfalls).

Improve this answer
$\endgroup$
2
  • $\begingroup$ can it be improved by entering position flags for input like ( w,1) (x,2) ( y,3)??? $\endgroup$
    – user8426627
    Jul 3, 2019 at 13:14
  • $\begingroup$ @user8426627 maybe. probably dpends on the data and problem. Im assuming what you mean, is if you add some form of positional embedding will it improve the model? Positional embeddings are used alot recently because popular attention-only architectures dont embed position inherently, while RNN's do. So in this case, youd hope its not necessary because of the LSTM, but you never know, adding that representation may assist in the training. Could be a fun experiment :) $\endgroup$
    – mshlis
    Jul 3, 2019 at 13:39
1
$\begingroup$

Shouldn't the hidden state of the RNN from the previous timestamps be enough?

It is theoretically enough to generate a sequence. However, allowing an input offers a couple of convenient extras:

  • Training data for output sequences is used twice - once as input (as previous sequence data), once as target (to establish loss metric). This may help training process as the decoder trains both as a decoder to the new sequence type and as a predictive model over the output sequence semi-independently - i.e. weights from input to RNN layer are affected by error gradients separately to weights between previous hidden state and next state, although the two sets of weights together influence output and next state, so are not fully independent over a sequence.

  • By allowing input of sequence so far generated, the decoder can work as a generator, where the next item in the sequence does not need to be the maximum probability item, but can be sampled or have rules applied. This allows for approaches such as BEAM search, commonly used in machine translation, which maintains several potential outputs, selecting best one at the end.

I have not done the experiment, but I suspect the first item results in faster and better generalisation. The second one is very convenient for natural language generation and similar problems.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I like to view it somewhat like while writing something from memory we frequently check what we have already written before so that it is consistent with our upcoming writing. $\endgroup$
    – user9947
    Jul 3, 2019 at 19:35
  • 1
    $\begingroup$ @DuttA: Yes, with that intuition, it can also be considered a form of regularisation, because the weights have to satisfy two constraints instead of one. $\endgroup$
    – Neil Slater
    Jul 3, 2019 at 19:37
0
$\begingroup$

In the original seq2seq paper, they used two RNN, one for encoding and one for decoding. In the encoder they need to unroll the inputs to capture the time dependency. Now if we want to pass the hidden state from the encoder to the decoder that means that the decoder hidden state shape needs to match the encoder (aka same architecture). Since the architecture is the same, we can not directly generate a sequence of n samples within the decoder without unrolling it and you can not unroll it without an input.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .