Artificial Intelligence Stack Exchange is a question and answer site for people interested in conceptual questions about life and challenges in a world where "cognitive" functions can be mimicked in purely digital environment. It only takes a minute to sign up.
Sign up to join this community
Why do we need both encoder and decoder in sequence to sequence prediction?
We could just have a single RNN that, given input $x$, outputs some value $y(t)$ and hidden state $h(t)$. Next, given $h(t)$ and $y(t)$, the next output $y(t+1)$ and hidden state $h(t+1)$ should be produced, and so on. The architecture shall consists of only one network instead of two separate ones.
(Old question, I know...)
It is not that we need both an encoder and decoder for sequence-to-sequence models - this decoupling of "reading" and "generating" just works better very often.
Example for Sequence-to-sequence without two RNNs
To prove my point above, here is an example from machine translation. Current machine translation systems are sequence-to-sequence models, and virtually all models have the bipartite structure of encoder and decoder.
Approaches like Eager Translation break this implied convention. They learn translation models that do not encode and decode with separate RNNs, but at every time step 1) read a source token and 2) produce a target token - with a single RNN.
Why encoder-decoder works better very often
Sequence-to-sequence modeling with encoder-decoder structure almost always implies attention in-between encoder and decoder. Attention relays information between the encoder and decoder, in the sense that every time the decoder has to generate the next item in the target sequence, an attention network computes a dynamic, useful "summary" of all encoder states.
This attention summary is different and recomputed for every decoding step. On the other hand, encoding the source sequence is done only once and then all encoder states are kept in memory.
The ability to have a direct view of the source sequence (using as a proxy the entire sequence of encoder states) via attention is what makes the encoder-decoder approach superioir to a single RNN.
In comparison, a single RNN only has a direct view on one element of the input sequence. Some interesting scenarios for a single RNN:
The usual purpose of using an encoder at the front end of a network and a decoder at the tail is to deal with features in between. Between the networks is a representation of the input with much of the redundancy removed.
Some of the approaches are discussed in On the Properties of Neural Machine Translation: Encoder–Decoder Approaches, 2014, Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, ,Yoshua Bengio. A single network that, given inputs, outputs some value and later appends that value to the input to produce the next output is not an equivalent circuit. Without the separation, network cannot be trained on the basis of some characteristic of the unique features found at the middle tap point.
Those learning how artificial networks can perform simple tasks tend to draw the system boundary around a sequence of layers where the inner layers are not manipulated differently from a forward propagation and back propagation perspective. They will call that the network and the layers that are not directly manipulated as elements in the series of layers are called hidden layers, which is correct, but they are not really hidden, they are just not addressed at the higher level, at the system boundary. They're manipulated as elements in the series by the forward and backward propagation mechanisms.
Those who have linked these sets of layers together into larger systems sometimes use the term network in the wider sense, a circuit that contains more than one sequence of layers. That's typical in NLP and motion control systems literature. The difference is the complexity of control of the learning process. Some control schemes require the interrelationship of data paths that are not at one of two ends.
Many have attempted to reduce this complexity, but there are control theory reasons why some systems cannot be reduced without losing control of some aspect of the learning process that is directly tied to the objectives of the system. Take a look at textual natural language translation, speech synthesis, robot imitation, automatic pilots, and terrain learners to see examples where the assembly of multiple networks into the larger system are necessary.
Also, feeding outputs into inputs defeats the design of the recurrent or long-short network design, which handles the retention of state in a way that converges to minimize loss inherently.
