How do I calculate the policy in the Proximal Policy Optimization algorithm?

Question

I recently watched the video on Proximal Policy Optimization (PPO). Now, I want to upgrade my actor-critic algorithm written in PyTorch with PPO, but I'am not sure how the new parameters / thetas are calculated.

In the paper Proximal Policy Optimization Algorithms (at page 5), the pseudocode of the PPO algorithm is shown:

It says to run $\pi_{\theta_{\text{old}}}$, compute advantage estimates and optimize the objective. But how can we calculate $\pi_\theta$ for the objective ratio, since we have not updated the $\pi_{\theta_{\text{old}}}$ yet?

Answer 1

You're right, the first time you run it the two policies ($\pi_{\theta old}$ and $\pi_\theta$) will be the same. This means your loss is simply the advantage (since you multiply the the ratio ($r(\theta)={\pi_\theta(a|s)\over\pi_{\theta old}(a|s)}$) by the advantage (so $loss=-r_t(\theta)A_t$).

However, with PPO you run multiple epochs of training on the same data. So after your first update you do the whole thing again (without exploring the environment any more) and this time $\pi_{\theta old}$ is different to $\pi_\theta$.

Here's a great explanation of the algorithm: https://stackoverflow.com/questions/46422845/what-is-the-way-to-understand-proximal-policy-optimization-algorithm-in-rl