6
$\begingroup$

The basic seq-2-seq model consists of 2 parts: a recurrent encoder that compresses a sequence to a vector and decoder that unrolls the vector into the output sequence:

enter image description here

Why is the output, w, x, y, z of the decoder used as its input? Shouldn't the hidden state of the RNN from the previous timestamps be enough?

| improve this question | |
$\endgroup$
3
$\begingroup$

In seq2seq they model the joint distribution of whatever char/word sequence by decomposing it into time-forward conditionals:
$\begin{align*} p(w_1,w_2,...,w_n) =& \ p(w_1)*p(w_2|w_1) * \ ... \ * p(w_n|w_1,...,w_{n-1}) \\ =& \ p(w_1)*\prod_{i=2}^{n}p(w_i|w_{<i}) \end{align*} $

This can be sampled by samping each of the conditional is ascending order. So thats exactly what theyre trying to imitate. You want the second output dependant on the sampled first output, not its distribution.

This is why the hidden state is NOT good for modeling this setup because it is a latent representation of the distribution, not a sample of the distribution.

Note: In training they use ground truth as input by default because its working under the assumption the model shouldve predicted the correct word and if it didnt the gradient of the word/char level loss will reflect that (this is called teacher forcing and has a multitude of pitfalls)

| improve this answer | |
$\endgroup$
  • $\begingroup$ can it be improved by entering position flags for input like ( w,1) (x,2) ( y,3)??? $\endgroup$ – user8426627 Jul 3 '19 at 13:14
  • $\begingroup$ @user8426627 maybe. probably dpends on the data and problem. Im assuming what you mean, is if you add some form of positional embedding will it improve the model? Positional embeddings are used alot recently because popular attention-only architectures dont embed position inherently, while RNN's do. So in this case, youd hope its not necessary because of the LSTM, but you never know, adding that representation may assist in the training. Could be a fun experiment :) $\endgroup$ – mshlis Jul 3 '19 at 13:39
1
$\begingroup$

Shouldn't the hidden state of the RNN from the previous timestamps be enough?

It is theoretically enough to generate a sequence. However, allowing an input offers a couple of convenient extras:

  • Training data for output sequences is used twice - once as input (as previous sequence data), once as target (to establish loss metric). This may help training process as the decoder trains both as a decoder to the new sequence type and as a predictive model over the output sequence semi-independently - i.e. weights from input to RNN layer are affected by error gradients separately to weights between previous hidden state and next state, although the two sets of weights together influence output and next state, so are not fully independent over a sequence.

  • By allowing input of sequence so far generated, the decoder can work as a generator, where the next item in the sequence does not need to be the maximum probability item, but can be sampled or have rules applied. This allows for approaches such as BEAM search, commonly used in machine translation, which maintains several potential outputs, selecting best one at the end.

I have not done the experiment, but I suspect the first item results in faster and better generalisation. The second one is very convenient for natural language generation and similar problems.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ I like to view it somewhat like while writing something from memory we frequently check what we have already written before so that it is consistent with our upcoming writing. $\endgroup$ – DuttaA Jul 3 '19 at 19:35
  • 1
    $\begingroup$ @DuttA: Yes, with that intuition, it can also be considered a form of regularisation, because the weights have to satisfy two constraints instead of one. $\endgroup$ – Neil Slater Jul 3 '19 at 19:37
0
$\begingroup$

In the original seq2seq paper, they used two RNN, one for encoding and one for decoding. In the encoder they need to unroll the inputs to capture the time dependency. Now if we want to pass the hidden state from the encoder to the decoder that means that the decoder hidden state shape needs to match the encoder (aka same architecture). Since the architecture is the same, we can not directly generate a sequence of n samples within the decoder without unrolling it and you can not unroll it without an input.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.