I'm trying to wrap my head around using LSTM in an RL algorithm like actor-critic or PPO. I've found this Github code which presents this in a very simple manner, however I have a very limited understanding of LSTM.
In each step the algorithm creates an entry of the following:
- previous state $s$;
- action selected $a$;
- reward $r$;
- next state $s'$;
- probability of action $prob$;
- hidden state before the step $h_{in}$ (in pytorch this is a 2-tuple containing the real hidden state and a cell state, but I'm not sure if this is relevant here);
- hidden state after the step $h_{out}$;
- boolean value to determine the end of an episode $done$.
This step is done until either the end of an episode or until the batch of entry reaches a certain size (here T_horizon). The resulting batch would look like this (I'm only keeping the more relevant variables): $(s_0, a_0, s1, h_0, h_1), (s_1, a_1, s_2, h_1, h_2) \ldots, (s_{T-1}, a_{T-1}, s_T, h_{T-1}, h_{T})$.
To make an update and train the net, the RL algorithm needs to compute for example the value function in state $s$, where $s$ is a vector of $\{s_0, \ldots, s_{T-1}\}$. However, the value function also needs a hidden state for the LSTM, and the code here uses $h_0$ to compute $V(s, h_0)$ and $h_1$ to compute $V(s', h_1)$. (We can see that the make_batch function returns only the first $h_{in}(=h_0)$ and first $h_{out}(=h_1)$).
So the question is, why can we use $h_0$ to calculate a value of another state (eg.: $V(s_8, h_0)$)? Why not $h_T$ and $h_{T-1}$? Or does it really matter? Can I use $(h_0, h_T)$ for hidden states instead of $(h_0, h_1)$? Shouldn't I use the corresponding $h_i$ for state $s_i$?
