4
$\begingroup$

Why do we need both encoder and decoder in sequence to sequence prediction?

We could just have a single RNN that, given input $x$, outputs some value $y(t)$ and hidden state $h(t)$. Next, given $h(t)$ and $y(t)$, the next output $y(t+1)$ and hidden state $h(t+1)$ should be produced, and so on. The architecture shall consists of only one network instead of two separate ones.

Improve this question
$\endgroup$
6
  • 1
    $\begingroup$ Could you maybe draw a diagram of your suggested RNN architecture? It is not 100% clear to me what you mean.The encoder/decoder architecture used in seq2seq language translation already uses a feedback loop from output to input in the decoder. I think you are asking why you cannot use the same network/parameters for both stages? But I'd like to see a picture or something more concrete in order to understand what you are proposing $\endgroup$
    – Neil Slater
    Commented Dec 3, 2018 at 16:55
  • $\begingroup$ Precisely, why not just use one network? I updated the original post by describing what I mean $\endgroup$
    – greensquare
    Commented Dec 3, 2018 at 20:14
  • $\begingroup$ Sorry I still don't get your proposed architecture. You don't give indication of time steps for x, which is also a sequence. Are you suggesting that you run the network just like a seq2seq network, where you feed in a sequence of $x$ (ignoring the predictions), then after some point - perhaps after an end token - start reading $y$ and feeding that in to the same inputs as $x$? What if $x$ and $y$ are sequences of different things? Are you proposing to input "blank" $y$ whilst feeding in $x$, then blank $x$ when feeding in $y$? $\endgroup$
    – Neil Slater
    Commented Dec 3, 2018 at 22:10
  • $\begingroup$ I am assuming batch size of 1. So, you feed in x1 (the first training sample), ignore the output and just keep the last hidden state of the RNN. Then, you run a loop that goes up to some max_len value and starts producing the output values based on the last hidden state and prev output value. Then you repeat the process for the next training sample x2. x is sequence of numbers and y(t) (at single time step) is just a number. Hope this clears things up $\endgroup$
    – greensquare
    Commented Dec 4, 2018 at 7:41
  • $\begingroup$ In your case do $x$ and $y$ represent the same kind of measurement? Typically you would use a seq2seq model if that was not the case (e.g. English and French words are not the same data type). However, you might use your proposed architecture if $x$ and $y$ were from the same series and your goal is to predict a continuation of the series - e.g. a financial series or NLP language model of a single language. $\endgroup$
    – Neil Slater
    Commented Dec 4, 2018 at 10:16

1 Answer 1

Reset to default
3
$\begingroup$

(Old question, I know...)

It is not that we need both an encoder and decoder for sequence-to-sequence models - this decoupling of "reading" and "generating" just works better very often.

Example for Sequence-to-sequence without two RNNs

To prove my point above, here is an example from machine translation. Current machine translation systems are sequence-to-sequence models, and virtually all models have the bipartite structure of encoder and decoder.

Approaches like Eager Translation break this implied convention. They learn translation models that do not encode and decode with separate RNNs, but at every time step 1) read a source token and 2) produce a target token - with a single RNN.

Why encoder-decoder works better very often

Sequence-to-sequence modeling with encoder-decoder structure almost always implies attention in-between encoder and decoder. Attention relays information between the encoder and decoder, in the sense that every time the decoder has to generate the next item in the target sequence, an attention network computes a dynamic, useful "summary" of all encoder states.

This attention summary is different and recomputed for every decoding step. On the other hand, encoding the source sequence is done only once and then all encoder states are kept in memory.

The ability to have a direct view of the source sequence (using as a proxy the entire sequence of encoder states) via attention is what makes the encoder-decoder approach superioir to a single RNN.

In comparison, a single RNN only has a direct view on one element of the input sequence. Some interesting scenarios for a single RNN:

  • At every time step, read one source token, then write one target token: Previous elements in the source sequence are represented only in lossy recurrent states, while future elements cannot be accessed at all.
  • First read all source tokens, then write all target tokens.: the meaning of the entire source sentence has to be compressed into a fixed-size recurrent state vector.
Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .