I'm trying to wrap my head around using LSTM in an RL algorithm like actor-critic or PPO. I've found this Github code which presents this in a very simple manner, however I have a very limited understanding of LSTM.

In each step the algorithm creates an entry of the following:

  • previous state $s$;
  • action selected $a$;
  • reward $r$;
  • next state $s'$;
  • probability of action $prob$;
  • hidden state before the step $h_{in}$ (in pytorch this is a 2-tuple containing the real hidden state and a cell state, but I'm not sure if this is relevant here);
  • hidden state after the step $h_{out}$;
  • boolean value to determine the end of an episode $done$.

This step is done until either the end of an episode or until the batch of entry reaches a certain size (here T_horizon). The resulting batch would look like this (I'm only keeping the more relevant variables): $(s_0, a_0, s1, h_0, h_1), (s_1, a_1, s_2, h_1, h_2) \ldots, (s_{T-1}, a_{T-1}, s_T, h_{T-1}, h_{T})$.

To make an update and train the net, the RL algorithm needs to compute for example the value function in state $s$, where $s$ is a vector of $\{s_0, \ldots, s_{T-1}\}$. However, the value function also needs a hidden state for the LSTM, and the code here uses $h_0$ to compute $V(s, h_0)$ and $h_1$ to compute $V(s', h_1)$. (We can see that the make_batch function returns only the first $h_{in}(=h_0)$ and first $h_{out}(=h_1)$).

So the question is, why can we use $h_0$ to calculate a value of another state (eg.: $V(s_8, h_0)$)? Why not $h_T$ and $h_{T-1}$? Or does it really matter? Can I use $(h_0, h_T)$ for hidden states instead of $(h_0, h_1)$? Shouldn't I use the corresponding $h_i$ for state $s_i$?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.