2
$\begingroup$

I recently watched the video on Proximal Policy Optimization (PPO). Now, I want to upgrade my actor-critic algorithm written in PyTorch with PPO, but I'am not sure how the new parameters / thetas are calculated.

In the paper Proximal Policy Optimization Algorithms (at page 5), the pseudocode of the PPO algorithm is shown:

enter image description here

It says to run $\pi_{\theta_{\text{old}}}$, compute advantage estimates and optimize the objective. But how can we calculate $\pi_\theta$ for the objective ratio, since we have not updated the $\pi_{\theta_{\text{old}}}$ yet?

$\endgroup$
2
$\begingroup$

You're right, the first time you run it the two policies ($\pi_{\theta old}$ and $\pi_\theta$) will be the same. This means your loss is simply the advantage (since you multiply the the ratio ($r(\theta)={\pi_\theta(a|s)\over\pi_{\theta old}(a|s)}$) by the advantage (so $loss=-r_t(\theta)A_t$).

However, with PPO you run multiple epochs of training on the same data. So after your first update you do the whole thing again (without exploring the environment any more) and this time $\pi_{\theta old}$ is different to $\pi_\theta$.

Here's a great explanation of the algorithm: https://stackoverflow.com/questions/46422845/what-is-the-way-to-understand-proximal-policy-optimization-algorithm-in-rl

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.