This is my own understanding of the hidden state in a recurrent network. If it's wrong, please, feel free to let me know.
Let's consider the following two input and output sequences
\begin{align}
X &= [a, b, c, d, \dots,y , z]\\
Y &= [b, c, d, e, \dots,z , a]
\end{align}
We will first try to train a multi-layer perceptron (MLP) with one input and one output from $X$ and $Y$. Here, the details of the hidden layers don't matter.
We can write this relationship in maths as
$$f(x)\rightarrow y$$
where $x$ is an element of $X$ and $y$ is an element of $Y$ and $f(\cdot)$ is our MLP.
After training, if given the input $a = x$, our neural network will give an output $b = y$ because $f(\cdot)$ learned the mapping between the sequence $X$ and $Y$.
Now, instead of the above sequences, try to teach the following sequences to the same MLP.
\begin{align}
X &= [a,a,b,b,c,c,\cdots, y,z,z]\\
Y &= [a,b,c,\cdots, z,a,b,c, \cdots, y,z]
\end{align}
More than likely, this MLP will not be able to learn the relationship between $X$ and $Y$. This is because a simple MLP can't learn and understand the relationship between the previous and current characters.
Now, we use the same sequences to train an RNN. In an RNN, we take two inputs, one for our input and the previous hidden values, and two outputs, one for the output and the next hidden values.
$$f(x, h_t)\rightarrow (y, h_{t+1})$$
Important: here $h_{t+1}$ represents the next hidden value.
We will execute some sequences of this RNN model. We initialize the hidden value to zero.
x = a and h = 0
(a,next_hidden) <- f(x,h)
prev_hidden = next_hidden
x = a and h = prev_hidden
(b,next_hidden) <- f(x,h)
prev_hidden = next_hidden
x = b and h = prev_hidden
(c,next_hidden) <- f(x,h)
prev_hidden = next_hidden
and so on
If we look at the above process we can see that we are taking the previous hidden state values to compute the next hidden state. What happens is while we iterate through this process prev_hidden = next_hidden
it also encodes some information about our sequence which will help in predicting our next character.