Where does the proximal policy optimization objective's ratio term come from?

Question

I will use the notation used in the proximal policy optimization paper.

What approximation is needed to arrive at the surrogate objective (equation (6) above) with the ratio $r_t(\theta)$?

Put another way, we start with the vanilla policy gradient objective and aim to optimize $L^\text{PG} (\theta) = \mathbb{E}_t \left [ \log \pi_\theta(a_t|s_t)A_t \right]$. Where in $L^\text{PG}(\theta)$ do I make an approximation to derive equation (6)?

I could not find this objective following the reference KL02.

Answer 1

No approximation here. The formula for $L^{CPI}$ can be directly derived from the formula for $L^{PG}$.

\begin{align} \nabla L^{PG} &= \mathbb{E}_{a_t, s_t \sim \pi_\theta} \Big[ \nabla \log \pi_\theta(a_t | s_t) A_t \Big] \\ &= \int_{a, s} \pi_\theta(a | s) \nabla \log \pi_\theta(a|s) A_t \\ &= \int_{a, s} \frac{\pi_{\theta_{old}}(a | s)}{\pi_{\theta_{old}}(a | s)} \pi_\theta(a | s) \frac{\nabla \pi_\theta(a | s)}{\pi_\theta(a | s)} A_t \\ &= \mathbb{E}_{a_t, s_t \sim \pi_{\theta_{old}}} \Big[ \frac{\nabla \pi_\theta(a_s | s_t)}{\pi_{\theta_{old}} (a_t | s_t)} A_t \Big] \\ &= \nabla L^{CPI} \end{align}

Now: \begin{align} L^{PG} &= \int \nabla L^{PG} = \mathbb{E}_{a_t, s_t \sim \pi_\theta} \Big[ \log \pi_\theta(a_t | s_t) A_t \Big] \\ L^{CPI} &= \int \nabla L^{CPI} = \mathbb{E}_{a_t, s_t \sim \pi_{\theta_{old}}} \Big[ \frac{\pi_\theta(a_s | s_t)}{\pi_{\theta_{old}} (a_t | s_t)} A_t \Big] \end{align}

The difference between the two is that in $L^{PG}$ states and actions are sampled using the original policy $\pi_\theta$. This means that once you perform one gradient update step you have to throw away the data and collect new.
In $L^{CPI}$ the states and actions are collected under $\pi_{\theta_{old}}$, which means that you can perform multiple updates with the same dataset. Obviously, if you perform only one update step with PPO you will get the vanilla PG. After the first update step the two objectives will differ.

There is a bit more theory regarding why you want to clip, but since this is not part of the question I will just refer you to this blog post that I wrote:
https://pi-tau.github.io/posts/actor-critic/#ppo

Answer 2

The ratio term appeared between Eq. (13)&(14) in Sch+15b, as a result of importance sampling:

$$ L^{\theta^{\prime}}(\theta)=E_{\left(s_t, a_t\right) \sim \pi_{\theta^{\prime}}}\left[\frac{\pi_\theta\left(a_t | s_t\right)}{\pi_{\theta^{\prime}}\left(a_t | s_t\right)} A^{\theta^{\prime}}\left(s_t, a_t\right)\right]. $$