2
$\begingroup$

I will use the notation used in the proximal policy optimization paper.

conservative policy iteration objective

What approximation is needed to arrive at the surrogate objective (equation (6) above) with the ratio $r_t(\theta)$?

Put another way, we start with the vanilla policy gradient objective and aim to optimize $L^\text{PG} (\theta) = \mathbb{E}_t \left [ \log \pi_\theta(a_t|s_t)A_t \right]$. Where in $L^\text{PG}(\theta)$ do I make an approximation to derive equation (6)?

I could not find this objective following the reference KL02.

$\endgroup$
1
  • $\begingroup$ I am not sure either. It doesn’t make sense. Im confused because its not symmetric(unclipped). If the advantage is positive and the ratio is greater then 1, you get a big positive gradient. But if you have a negative advantage and a ratio less then 1(same scenario but flipped in my mind), then you get a small negative gradient. In my mind, given the same advantage magnitude, opposite signs, the gradient of a ratio of 2 would be the negative of a gradient of a ratio of 0.5. Does that make sense? $\endgroup$
    – Yuh
    May 11, 2023 at 7:13

2 Answers 2

0
$\begingroup$

The ratio term appeared between Eq. (13)&(14) in Sch+15b, as a result of importance sampling:

$$ L^{\theta^{\prime}}(\theta)=E_{\left(s_t, a_t\right) \sim \pi_{\theta^{\prime}}}\left[\frac{\pi_\theta\left(a_t | s_t\right)}{\pi_{\theta^{\prime}}\left(a_t | s_t\right)} A^{\theta^{\prime}}\left(s_t, a_t\right)\right]. $$

$\endgroup$
0
$\begingroup$

No approximation here. The formula for $L^{CPI}$ can be directly derived from the formula for $L^{PG}$.

\begin{align} \nabla L^{PG} &= \mathbb{E}_{a_t, s_t \sim \pi_\theta} \Big[ \nabla \log \pi_\theta(a_t | s_t) A_t \Big] \\ &= \int_{a, s} \pi_\theta(a | s) \nabla \log \pi_\theta(a|s) A_t \\ &= \int_{a, s} \frac{\pi_{\theta_{old}}(a | s)}{\pi_{\theta_{old}}(a | s)} \pi_\theta(a | s) \frac{\nabla \pi_\theta(a | s)}{\pi_\theta(a | s)} A_t \\ &= \mathbb{E}_{a_t, s_t \sim \pi_{\theta_{old}}} \Big[ \frac{\nabla \pi_\theta(a_s | s_t)}{\pi_{\theta_{old}} (a_t | s_t)} A_t \Big] \\ &= \nabla L^{CPI} \end{align}

Now: \begin{align} L^{PG} &= \int \nabla L^{PG} = \mathbb{E}_{a_t, s_t \sim \pi_\theta} \Big[ \log \pi_\theta(a_t | s_t) A_t \Big] \\ L^{CPI} &= \int \nabla L^{CPI} = \mathbb{E}_{a_t, s_t \sim \pi_{\theta_{old}}} \Big[ \frac{\pi_\theta(a_s | s_t)}{\pi_{\theta_{old}} (a_t | s_t)} A_t \Big] \end{align}

The difference between the two is that in $L^{PG}$ states and actions are sampled using the original policy $\pi_\theta$. This means that once you perform one gradient update step you have to throw away the data and collect new.
In $L^{CPI}$ the states and actions are collected under $\pi_{\theta_{old}}$, which means that you can perform multiple updates with the same dataset. Obviously, if you perform only one update step with PPO you will get the vanilla PG. After the first update step the two objectives will differ.

There is a bit more theory regarding why you want to clip, but since this is not part of the question I will just refer you to this blog post that I wrote:
https://pi-tau.github.io/posts/actor-critic/#ppo

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .