I'm working on research rn using LSTM as an encoder decoder in hopes to make inferences. The reason we are using encoder decoder for this is because there is hopes that the hidden state given by the encoder when put into the decoder will allow the LSTM to make insights into potential latents. Does this make sense? I'm a bit fuzzy on this because I really don't know what the hidden state is and only the LSTM really understands it, but we're using separate LSTMs for the encoder and decoder and I can't see how the hidden state from the encoder LSTM can be useful to the decoder LSTM because only the encoder LSTM really understands it.


This is my own understanding of hidden state in a recurrent network and if its wrong please feel free to let me know.

Lets take this simple sequence first,

   X = [a,b,c,d,.......,y,z]
   Y = [b,c,d,e,.......,z,a]

Instead of RNN we will first try to train this in a simple multi layer neural network with one input and one output, here hidden layers details doesn't matter.

We can write this relationship in maths as

$f(x)\rightarrow y$

where x is an element of X and y is an element of Y and f() is our neural network.

After training, if given input x = a our neural network will give an output b because f() learned the mapping between the sequence X and Y.

Now instead of the above sequence try teach this sequence to the same neural network.

 X = [a,a,b,b,c,c,.........y,z,z]
 Y = [a,b,c,,...z,a,b,c......y,z]

More than likely this neural netwok will not be able to learn the relationship between X and Y. This is because a simple neural network can't learn and understand the relationship between the previous and current characters.

Now we train the same sequence in RNN, in RNN we take two inputs one for our input and a previous hidden values and two outputs one for output and next hidden values

$f(x,h)\rightarrow (y,h+t)$

Important: here h+t represents next hidden value and not the arithematic addition with time step t.

We will execute some sequence of this RNN model, at the start hidden values are considerd as zeros

   x = a and h = 0
   f(x,h) = (a,next_hidden)
   prev_hidden = next_hidden

   x = a and h = prev_hidden
   f(x,h) = (b,next_hidden)
   prev_hidden = next_hidden

   x = b and h = prev_hidden
   f(x,h) = (c,next_hidden)
   prev_hidden = next_hidden

   and so on 

If we look at the above process we can see that we are taking previous hidden state values to compute the next hidden state. What happens is while we iterate through this process prev_hidden = next_hidden it also encodes some information about our sequence which will help in predicting our next character.


As you said, one way to look at it is definitely that the LSTM-encoder's encoding can be only understood by itself, that's why the decoder exists there. An optimisation process encoded it, why couldn't an optimisation process decode it?

The hidden state is essentially just an encoding of the information you gave it keeping the time-dependencies in check. Most encoder-decoder networks are trained end to end meaning, when the encoding is learned a corresponding decoding is learned simultaneously to decode the encoded latent in your desired format.

I'd recommend you read this blog on how transformer models are used to convert French to English, as it would give you better intuition and understanding on what happens with encoder-decoder sequence models


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.