# Which hidden state should I use for a trajectory when incorporating LSTM into RL?

I'm trying to wrap my head around using LSTM in an RL algorithm like actor-critic or PPO. I've found this Github code which presents this in a very simple manner, however I have a very limited understanding of LSTM.

In each step the algorithm creates an entry of the following:

• previous state $$s$$;
• action selected $$a$$;
• reward $$r$$;
• next state $$s'$$;
• probability of action $$prob$$;
• hidden state before the step $$h_{in}$$ (in pytorch this is a 2-tuple containing the real hidden state and a cell state, but I'm not sure if this is relevant here);
• hidden state after the step $$h_{out}$$;
• boolean value to determine the end of an episode $$done$$.

This step is done until either the end of an episode or until the batch of entry reaches a certain size (here T_horizon). The resulting batch would look like this (I'm only keeping the more relevant variables): $$(s_0, a_0, s1, h_0, h_1), (s_1, a_1, s_2, h_1, h_2) \ldots, (s_{T-1}, a_{T-1}, s_T, h_{T-1}, h_{T})$$.

To make an update and train the net, the RL algorithm needs to compute for example the value function in state $$s$$, where $$s$$ is a vector of $$\{s_0, \ldots, s_{T-1}\}$$. However, the value function also needs a hidden state for the LSTM, and the code here uses $$h_0$$ to compute $$V(s, h_0)$$ and $$h_1$$ to compute $$V(s', h_1)$$. (We can see that the make_batch function returns only the first $$h_{in}(=h_0)$$ and first $$h_{out}(=h_1)$$).

So the question is, why can we use $$h_0$$ to calculate a value of another state (eg.: $$V(s_8, h_0)$$)? Why not $$h_T$$ and $$h_{T-1}$$? Or does it really matter? Can I use $$(h_0, h_T)$$ for hidden states instead of $$(h_0, h_1)$$? Shouldn't I use the corresponding $$h_i$$ for state $$s_i$$?