I understand how to derive the vanilla policy gradient $$ \begin{align} \nabla_{\theta}J(\pi_{\theta}) = \mathbb{E}_{\pi_{\theta}} \left[ \sum_{t = 0}^{T} \nabla_{\theta} \log \pi_{\theta}(a_{t} \mid s_{t}) \hat{A}^{\pi_{\theta}}(s_{t}, a_{t}) \right] \end{align} $$ as is also demonstrated in the openai spinning up documentation. Reading the PPO paper, they say that the most commonly used gradient estimator is $$ \begin{align} \nabla_{\theta}J(\pi_{\theta}) = \mathbb{E}_{\pi_{\theta}} \left[\nabla_{\theta}\log \pi_{\theta}(a_{t} \mid s_{t}) \hat{A}^{\pi_{\theta}}(s_{t}, a_{t}) \right] \end{align} $$ and from there they argue that this is equivalent to using the loss function $$ \begin{align} \mathcal{L}^{\text{PG}}(\theta) = \mathbb{E}_{\pi_{\theta}} \left[\log \pi_{\theta}(a_{t} \mid s_{t}) \hat{A}^{\pi_{\theta}}(s_{t}, a_{t}) \right] \end{align} $$ where the derivative is then taken with respect to $\theta$.

Question: How does the vanilla policy gradient relate to the gradient stated in the PPO paper? More precisely, they seem to be identical up to the sum over $t$. Why can the sum be ignored and the gradient is still the same?

Thanks in advance for any help!


1 Answer 1


Short answer:

The expectation $\mathbb{E}_t$ in the PPO paper is not an expectation over trajectories, but a mean. I suppose the confusion comes from there. The two quantities are otherwise very similar. The gradient is however not the same, as it is a biased estimator.

Long answer:

There are many related formulations for RL problems, on one hand we have the finite-horizon (episodic) setting:

$$ J_1(\theta)=\mathbb{E}_{\tau\sim p_{\pi_{\theta}}}\left[\sum_{t= 0}^T r(s_t,a_t)\right]. $$

On the other hand, we have the commonly used infinite-horizon setting (assumed by the PPO paper):

$$ J_2(\theta)=\mathbb{E}_{\tau\sim p_{\pi_{\theta}}}\left[\sum_{t\geq 0} \gamma^tr(s_t,a_t)\right]. $$

In the finite horizon setting, we have :

\begin{equation} \nabla_{\theta} J_1(\theta) = \mathbb{E}_{\tau\sim p_{\pi_{\theta}}}\left[ \left(\sum_{t= 0}^T \nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) \right)R(\tau) \right], \end{equation}

where $R(\tau)=\sum_{t= 0}^T r(s_t,a_t)$. As explained in the SpinningUp post, we can replace $R(\tau)$ by $A_{\pi_{\theta}}(s_t,a_t)$.

Now, a similar proof shows that:

\begin{equation} \nabla_{\theta} J_2(\theta) = \mathbb{E}_{\tau\sim p_{\pi_{\theta}}}\left[ \sum_{t\geq 0}\gamma^t \nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) A_{\pi_{\theta}}(s_t,a_t) \right]. \end{equation}

Now, in practice it is very common to approximate

$$\nabla_{\theta} J_2(\theta) \approx \mathbb{E}_{\tau\sim p_{\pi_{\theta}}}\left[ \sum_{t\geq 0}\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) A_{\pi_{\theta}}(s_t,a_t) \right]:=g^{\gamma},$$

that is we drop the discount factor $\gamma^t$. This relates to the policy gradient theorem and that we want to sample states from the steady-state distribution instead of the discounted state distribution.

Now, we want to approximate $A_{\pi}(s_t,a_t)$ by an estimator $\hat{A_t}$. Schulman et al in the GAE paper introduce the definition of a $\gamma$-just estimator. The exact definition is maybe not essential, but $\gamma$-just estimators include $Q_{\pi}(s_t,a_t)$, $A_{\pi}(s_t,a_t)$ and the GAE estimator used in the PPO paper.

So, if $\hat{A}_t$ is $\gamma$-just, then we have: $$g^{\gamma} = \mathbb{E}_{\tau\sim p_{\pi_{\theta}}}\left[ \sum_{t\geq 0}\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) \hat{A}_t \right]. $$

This expectation is however still not convenient for a practical algorithm, as we need to collect the trajectories with a policy with older weights $\pi_{\theta_{\text{old}}}$. So we approximate

$$g^{\gamma} \approx \mathbb{E}_{\tau\sim p_{\pi_{\theta_{\text{old}}}}}\left[ \sum_{t\geq 0}\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) \hat{A}_t \right]:=\hat{g}. $$

Now, we need to approximate this expectation, so we collect $N$ rollouts of identical length $T$ (for simplicity) with the policy $\pi_{\theta_{\text{old}}}$, then we have an approximation of $g^{\gamma}$:

$$\nabla_{\theta} J_2(\theta) \approx g^{\gamma}\approx \hat{g}\approx\frac{1}{N}\sum_{n=0}^N\sum_{t=0}^T \nabla_{\theta}\log\pi_{\theta}(a_t^n\mid s_t^n) \hat{A}_t^n,$$ which is what the $\mathbb{E}_t$ in the PPO paper means. The expression on the right-hand side is now finally useful for gradient descent-like algorithms, as $\hat{A_t}$does not depend on $\theta$.

Note also that much of the PPO paper aims to ensure that the last approximation by replacing $p_{\pi_{\theta}}$ by $p_{\pi_{\theta_{\text{old}}}}$ is valid.

  • $\begingroup$ Thanks for the detailed explanation. Indeed, it does seem like the confusion stems from the $\hat{E}_{t}$ notation. I understand your derivation but I dont get why your last estimator $\hat{g}$ is what they refer to by $\hat{E}_{t}$. For me "an empirical batch over a finite batch of samples" is rather your last estimor without the sum over t, i.e. $\frac{1}{N} \sum_{n = 0}^{N} \nabla_{\theta} \log \pi_{\theta}(a_{t}^{n} \mid s_{t}^{n}) \hat{A}_{t}^{n}$. $\endgroup$
    – Peter
    Jul 8, 2022 at 16:25
  • $\begingroup$ We want to estimate $\hat{g}$, which is an expectation over trajectories. Depending on the algorithm prior to PPO, this is done differently. It relates to the discussion of TD(0) vs n-step TD estimation (although the quantity in the expectation is different, the principle is similar). What you propose in the comment is to take $n$ samples of 1-step tranistions, which is often done in off-policy methods (like DDPG). The other way, done in VPG is to collect one trajectory of length $T$. You can also take $n$ trajectories of length $T$ as in A3C. The A3C paper shows many ways to estimate it. $\endgroup$
    – matorbi
    Jul 9, 2022 at 18:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .