# Why do we need both encoder and decoder in sequence to sequence prediction?

Why do we need both encoder and decoder in sequence to sequence prediction?

We could just have a single RNN that, given input $$x$$, outputs some value $$y(t)$$ and hidden state $$h(t)$$. Next, given $$h(t)$$ and $$y(t)$$, the next output $$y(t+1)$$ and hidden state $$h(t+1)$$ should be produced, and so on. The architecture shall consists of only one network instead of two separate ones.

• Could you maybe draw a diagram of your suggested RNN architecture? It is not 100% clear to me what you mean.The encoder/decoder architecture used in seq2seq language translation already uses a feedback loop from output to input in the decoder. I think you are asking why you cannot use the same network/parameters for both stages? But I'd like to see a picture or something more concrete in order to understand what you are proposing – Neil Slater Dec 3 '18 at 16:55
• Precisely, why not just use one network? I updated the original post by describing what I mean – greensquare Dec 3 '18 at 20:14
• Sorry I still don't get your proposed architecture. You don't give indication of time steps for x, which is also a sequence. Are you suggesting that you run the network just like a seq2seq network, where you feed in a sequence of $x$ (ignoring the predictions), then after some point - perhaps after an end token - start reading $y$ and feeding that in to the same inputs as $x$? What if $x$ and $y$ are sequences of different things? Are you proposing to input "blank" $y$ whilst feeding in $x$, then blank $x$ when feeding in $y$? – Neil Slater Dec 3 '18 at 22:10
• I am assuming batch size of 1. So, you feed in x1 (the first training sample), ignore the output and just keep the last hidden state of the RNN. Then, you run a loop that goes up to some max_len value and starts producing the output values based on the last hidden state and prev output value. Then you repeat the process for the next training sample x2. x is sequence of numbers and y(t) (at single time step) is just a number. Hope this clears things up – greensquare Dec 4 '18 at 7:41
• In your case do $x$ and $y$ represent the same kind of measurement? Typically you would use a seq2seq model if that was not the case (e.g. English and French words are not the same data type). However, you might use your proposed architecture if $x$ and $y$ were from the same series and your goal is to predict a continuation of the series - e.g. a financial series or NLP language model of a single language. – Neil Slater Dec 4 '18 at 10:16

## 2 Answers

(Old question, I know...)

It is not that we need both an encoder and decoder for sequence-to-sequence models - this decoupling of "reading" and "generating" just works better very often.

Example for Sequence-to-sequence without two RNNs

To prove my point above, here is an example from machine translation. Current machine translation systems are sequence-to-sequence models, and virtually all models have the bipartite structure of encoder and decoder.

Approaches like Eager Translation break this implied convention. They learn translation models that do not encode and decode with separate RNNs, but at every time step 1) read a source token and 2) produce a target token - with a single RNN.

Why encoder-decoder works better very often

Sequence-to-sequence modeling with encoder-decoder structure almost always implies attention in-between encoder and decoder. Attention relays information between the encoder and decoder, in the sense that every time the decoder has to generate the next item in the target sequence, an attention network computes a dynamic, useful "summary" of all encoder states.

This attention summary is different and recomputed for every decoding step. On the other hand, encoding the source sequence is done only once and then all encoder states are kept in memory.

The ability to have a direct view of the source sequence (using as a proxy the entire sequence of encoder states) via attention is what makes the encoder-decoder approach superioir to a single RNN.

In comparison, a single RNN only has a direct view on one element of the input sequence. Some interesting scenarios for a single RNN:

• At every time step, read one source token, then write one target token: Previous elements in the source sequence are represented only in lossy recurrent states, while future elements cannot be accessed at all.
• First read all source tokens, then write all target tokens.: the meaning of the entire source sentence has to be compressed into a fixed-size recurrent state vector.

The usual purpose of using an encoder at the front end of a network and a decoder at the tail is to deal with features in between. Between the networks is a representation of the input with much of the redundancy removed.

Some of the approaches are discussed in On the Properties of Neural Machine Translation: Encoder–Decoder Approaches, 2014, Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, ,Yoshua Bengio. A single network that, given inputs, outputs some value and later appends that value to the input to produce the next output is not an equivalent circuit. Without the separation, network cannot be trained on the basis of some characteristic of the unique features found at the middle tap point.

Those learning how artificial networks can perform simple tasks tend to draw the system boundary around a sequence of layers where the inner layers are not manipulated differently from a forward propagation and back propagation perspective. They will call that the network and the layers that are not directly manipulated as elements in the series of layers are called hidden layers, which is correct, but they are not really hidden, they are just not addressed at the higher level, at the system boundary. They're manipulated as elements in the series by the forward and backward propagation mechanisms.

Those who have linked these sets of layers together into larger systems sometimes use the term network in the wider sense, a circuit that contains more than one sequence of layers. That's typical in NLP and motion control systems literature. The difference is the complexity of control of the learning process. Some control schemes require the interrelationship of data paths that are not at one of two ends.

Many have attempted to reduce this complexity, but there are control theory reasons why some systems cannot be reduced without losing control of some aspect of the learning process that is directly tied to the objectives of the system. Take a look at textual natural language translation, speech synthesis, robot imitation, automatic pilots, and terrain learners to see examples where the assembly of multiple networks into the larger system are necessary.

Also, feeding outputs into inputs defeats the design of the recurrent or long-short network design, which handles the retention of state in a way that converges to minimize loss inherently.

• Thanks, but what if you run episodes similar to policy gradient in reinforcement learning? Let's say you have episode of 10 runs and the goal is to have subsequence of [1,2,4] in a sequence of 10 numbers which range from 1 to 5. Let's say initial x =  (random number). Later policy grad. produces value  .. somewhere in the middle number 2 and 4 and so on up to 10 values. So the RNN has to remember all previous actions that were produced like [2,1,...,2,4,..] including initial state x. After the episode is finished, you start a new one. The only downside I see is batch size 1. – greensquare Dec 4 '18 at 7:57
• Sorry for the confusion, it's not appending the prev. value it just uses it in combination with the previous RNN hidden state similar to what the decoder does – greensquare Dec 4 '18 at 8:00
• Abbreviated explanations in comments do not usually illuminate new ideas very well. If you use the LaTeX capability of this site with clear explanations of each variable and the context in an algorithm or some diagrammed design, and then add that in a new question or clarify your existing one, we'll be able to provide more valuable feedback. – han_nah_han_ Dec 11 '18 at 11:36
• Alright, simple as this. I have dialogue between two users USR1 and USR2. Do you think dialogue between USR1 and USR2 can be modeled using one RNN where input to RNN are utterances from both USR1 and USR2 or should I use seq2seq where USR1 input goes to encoder and USR2 goes to decoder? – greensquare Jan 5 '19 at 9:56