I have a question regarding the functionality of the PPO2 algorithm together with the Stable Baselines implementation:

From the original paper I know that the policy parameters $\theta$ are updated K-times using the steps sampled (n_env * T steps):

enter image description here

When updating the policy parameters for a state $s_t$, are only the state observations $a_t$ and reward $r_{t+1}$ of this step considered, or also the state observations and rewards of the following steps ($t+1$) considered? My understanding is that the policy update with stochastic gradient ascent works just like in supervised learning.

I know that PPO2 uses a truncated TD($\lambda$) approach (T timesteps considered). So I guess that during the policy update for each state, subsequent states are only considered through the advantage function $A_t$ but not through the values of subsequent state observations and rewards themselves? Is that true?

I do not quite get the Stable Baselines implementation in the method _train_step() of the PPO2 implementation so therefore the question here.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.