# What exactly is a hidden state in an LSTM and RNN?

I'm working on a project, where we use an encoder-decoder architecture. We decided to use an LSTM for both the encoder and decoder due to its hidden states. In my specific case, the hidden state of the encoder is passed to the decoder, and this would allow the model to learn better latent representations.

Does this make sense?

I am a bit confused about this because I really don't know what the hidden state is. Moreover, we're using separate LSTMs for the encoder and decoder, so I can't see how the hidden state from the encoder LSTM can be useful to the decoder LSTM because only the encoder LSTM really understands it.

## 4 Answers

This is my own understanding of the hidden state in a recurrent network. If it's wrong, please, feel free to let me know.

Let's consider the following two input and output sequences

\begin{align} X &= [a, b, c, d, \dots,y , z]\\ Y &= [b, c, d, e, \dots,z , a] \end{align}

We will first try to train a multi-layer perceptron (MLP) with one input and one output from $$X$$ and $$Y$$. Here, the details of the hidden layers don't matter.

We can write this relationship in maths as

$$f(x)\rightarrow y$$

where $$x$$ is an element of $$X$$ and $$y$$ is an element of $$Y$$ and $$f(\cdot)$$ is our MLP.

After training, if given the input $$a = x$$, our neural network will give an output $$b = y$$ because $$f(\cdot)$$ learned the mapping between the sequence $$X$$ and $$Y$$.

Now, instead of the above sequences, try to teach the following sequences to the same MLP.

\begin{align} X &= [a,a,b,b,c,c,\cdots, y,z,z]\\ Y &= [a,b,c,\cdots, z,a,b,c, \cdots, y,z] \end{align}

More than likely, this MLP will not be able to learn the relationship between $$X$$ and $$Y$$. This is because a simple MLP can't learn and understand the relationship between the previous and current characters.

Now, we use the same sequences to train an RNN. In an RNN, we take two inputs, one for our input and the previous hidden values, and two outputs, one for the output and the next hidden values.

$$f(x, h_t)\rightarrow (y, h_{t+1})$$

Important: here $$h_{t+1}$$ represents the next hidden value.

We will execute some sequences of this RNN model. We initialize the hidden value to zero.

x = a and h = 0
f(x,h) = (a,next_hidden)
prev_hidden = next_hidden

x = a and h = prev_hidden
f(x,h) = (b,next_hidden)
prev_hidden = next_hidden

x = b and h = prev_hidden
f(x,h) = (c,next_hidden)
prev_hidden = next_hidden

and so on


If we look at the above process we can see that we are taking the previous hidden state values to compute the next hidden state. What happens is while we iterate through this process prev_hidden = next_hidden it also encodes some information about our sequence which will help in predicting our next character.

• I have a question regarding the output and hidden state here. here i already know what i output will look like at each time step then then should the hidden state be unknown? a normal neural net is farely simple i give it some inputs and corresponding outputs and it calculates the weigths and biases in order to predict a data set that you give it.. can i be using lstm to do the same thing? Sep 29 '21 at 12:19

As you said, one way to look at it is definitely that the LSTM-encoder's encoding can be only understood by itself, that's why the decoder exists there. An optimisation process encoded it, why couldn't an optimisation process decode it?

The hidden state is essentially just an encoding of the information you gave it keeping the time-dependencies in check. Most encoder-decoder networks are trained end to end meaning, when the encoding is learned a corresponding decoding is learned simultaneously to decode the encoded latent in your desired format.

I'd recommend you read this blog on how transformer models are used to convert French to English, as it would give you better intuition and understanding on what happens with encoder-decoder sequence models

I like to think of hidden states as intermediate representations of input within a neural system. The overall goal of the system is to re-represent an input in some specific way so that the system can produce some target output. Each layer within a neural network can only really "see" an input according to the specifics of its nodes, so each layer produces unique "snapshots" of whatever it is processing. Hidden states are sort of intermediate snapshots of the original input data, transformed in whatever way the given layer's nodes and neural weighting require.

The snapshots are just vectors so they can theoretically be processed by any other layer - by either an encoding layer or a decoding layer in your example.

The hidden state in a RNN is basically just like a hidden layer in a regular feed-forward network - it just happens to also be used as an additional input to the RNN at the next time step.

A simple RNN then might have an input $$x_t$$, a hidden layer $$h_t$$, and an output $$y_t$$ at each time step $$t$$. The values of the hidden layer $$h_t$$ are often computed as:

$$h_t = f(W_{xh}x_t + W_{hh}h_{t-1})$$

Where $$f$$ is some non-linear function, $$W_{xh}$$ is a weight matrix of size $$h\times x$$, and $$W_{hh}$$ is a weight matrix of size $$h\times h$$. I've left out the bias terms for simplicity.

Thus, the values of the hidden layer $$h_t$$ depend on the input $$x_t$$ as well as on the previous hidden state $$h_{t-1}$$ (literally, the previous values of the hidden layer).