I have a question regarding the functionality of the PPO2 algorithm together with the Stable Baselines implementation:

From the original paper I know that the policy parameters $\theta$ are updated K-times using the steps sampled (n_env * T steps):

enter image description here

When updating the policy parameters for a state $s_t$, are only the state observations $a_t$ and reward $r_{t+1}$ of this step considered, or also the state observations and rewards of the following steps ($t+1$) considered? My understanding is that the policy update with stochastic gradient ascent works just like in supervised learning.

I know that PPO2 uses a truncated TD($\lambda$) approach (T timesteps considered). So I guess that during the policy update for each state, subsequent states are only considered through the advantage function $A_t$ but not through the values of subsequent state observations and rewards themselves? Is that true?

I do not quite get the Stable Baselines implementation in the method _train_step() of the PPO2 implementation so therefore the question here.



You must log in to answer this question.