What exactly is a hidden state in an LSTM and RNN?

I'm working on a project, where we use an encoder-decoder architecture. We decided to use an LSTM for both the encoder and decoder due to its hidden states. In my specific case, the hidden state of the encoder is passed to the decoder, and this would allow the model to learn better latent representations.

Does this make sense?

I am a bit confused about this because I really don't know what the hidden state is. Moreover, we're using separate LSTMs for the encoder and decoder, so I can't see how the hidden state from the encoder LSTM can be useful to the decoder LSTM because only the encoder LSTM really understands it.

This is my own understanding of hidden state in a recurrent network and if its wrong please feel free to let me know.

Lets take this simple sequence first,

   X = [a,b,c,d,.......,y,z]
Y = [b,c,d,e,.......,z,a]


Instead of RNN we will first try to train this in a simple multi layer neural network with one input and one output, here hidden layers details doesn't matter.

We can write this relationship in maths as

$$f(x)\rightarrow y$$

where x is an element of X and y is an element of Y and f() is our neural network.

After training, if given input x = a our neural network will give an output b because f() learned the mapping between the sequence X and Y.

Now instead of the above sequence try teach this sequence to the same neural network.

 X = [a,a,b,b,c,c,.........y,z,z]
Y = [a,b,c,,...z,a,b,c......y,z]


More than likely this neural netwok will not be able to learn the relationship between X and Y. This is because a simple neural network can't learn and understand the relationship between the previous and current characters.

Now we train the same sequence in RNN, in RNN we take two inputs one for our input and a previous hidden values and two outputs one for output and next hidden values

$$f(x,h)\rightarrow (y,h+t)$$

Important: here h+t represents next hidden value and not the arithematic addition with time step t.

We will execute some sequence of this RNN model, at the start hidden values are considerd as zeros

   x = a and h = 0
f(x,h) = (a,next_hidden)
prev_hidden = next_hidden

x = a and h = prev_hidden
f(x,h) = (b,next_hidden)
prev_hidden = next_hidden

x = b and h = prev_hidden
f(x,h) = (c,next_hidden)
prev_hidden = next_hidden

and so on


If we look at the above process we can see that we are taking previous hidden state values to compute the next hidden state. What happens is while we iterate through this process prev_hidden = next_hidden it also encodes some information about our sequence which will help in predicting our next character.

• I have a question regarding the output and hidden state here. here i already know what i output will look like at each time step then then should the hidden state be unknown? a normal neural net is farely simple i give it some inputs and corresponding outputs and it calculates the weigths and biases in order to predict a data set that you give it.. can i be using lstm to do the same thing? Sep 29 at 12:19

As you said, one way to look at it is definitely that the LSTM-encoder's encoding can be only understood by itself, that's why the decoder exists there. An optimisation process encoded it, why couldn't an optimisation process decode it?

The hidden state is essentially just an encoding of the information you gave it keeping the time-dependencies in check. Most encoder-decoder networks are trained end to end meaning, when the encoding is learned a corresponding decoding is learned simultaneously to decode the encoded latent in your desired format.

I'd recommend you read this blog on how transformer models are used to convert French to English, as it would give you better intuition and understanding on what happens with encoder-decoder sequence models