# Why can the sum over timesteps in the Vanilla Policy Gradient be ignored?

I understand how to derive the vanilla policy gradient \begin{align} \nabla_{\theta}J(\pi_{\theta}) = \mathbb{E}_{\pi_{\theta}} \left[ \sum_{t = 0}^{T} \nabla_{\theta} \log \pi_{\theta}(a_{t} \mid s_{t}) \hat{A}^{\pi_{\theta}}(s_{t}, a_{t}) \right] \end{align} as is also demonstrated in the openai spinning up documentation. Reading the PPO paper, they say that the most commonly used gradient estimator is \begin{align} \nabla_{\theta}J(\pi_{\theta}) = \mathbb{E}_{\pi_{\theta}} \left[\nabla_{\theta}\log \pi_{\theta}(a_{t} \mid s_{t}) \hat{A}^{\pi_{\theta}}(s_{t}, a_{t}) \right] \end{align} and from there they argue that this is equivalent to using the loss function \begin{align} \mathcal{L}^{\text{PG}}(\theta) = \mathbb{E}_{\pi_{\theta}} \left[\log \pi_{\theta}(a_{t} \mid s_{t}) \hat{A}^{\pi_{\theta}}(s_{t}, a_{t}) \right] \end{align} where the derivative is then taken with respect to $$\theta$$.

Question: How does the vanilla policy gradient relate to the gradient stated in the PPO paper? More precisely, they seem to be identical up to the sum over $$t$$. Why can the sum be ignored and the gradient is still the same?

Thanks in advance for any help!

The expectation $$\mathbb{E}_t$$ in the PPO paper is not an expectation over trajectories, but a mean. I suppose the confusion comes from there. The two quantities are otherwise very similar. The gradient is however not the same, as it is a biased estimator.

There are many related formulations for RL problems, on one hand we have the finite-horizon (episodic) setting:

$$J_1(\theta)=\mathbb{E}_{\tau\sim p_{\pi_{\theta}}}\left[\sum_{t= 0}^T r(s_t,a_t)\right].$$

On the other hand, we have the commonly used infinite-horizon setting (assumed by the PPO paper):

$$J_2(\theta)=\mathbb{E}_{\tau\sim p_{\pi_{\theta}}}\left[\sum_{t\geq 0} \gamma^tr(s_t,a_t)\right].$$

In the finite horizon setting, we have :

$$\begin{equation} \nabla_{\theta} J_1(\theta) = \mathbb{E}_{\tau\sim p_{\pi_{\theta}}}\left[ \left(\sum_{t= 0}^T \nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) \right)R(\tau) \right], \end{equation}$$

where $$R(\tau)=\sum_{t= 0}^T r(s_t,a_t)$$. As explained in the SpinningUp post, we can replace $$R(\tau)$$ by $$A_{\pi_{\theta}}(s_t,a_t)$$.

Now, a similar proof shows that:

$$\begin{equation} \nabla_{\theta} J_2(\theta) = \mathbb{E}_{\tau\sim p_{\pi_{\theta}}}\left[ \sum_{t\geq 0}\gamma^t \nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) A_{\pi_{\theta}}(s_t,a_t) \right]. \end{equation}$$

Now, in practice it is very common to approximate

$$\nabla_{\theta} J_2(\theta) \approx \mathbb{E}_{\tau\sim p_{\pi_{\theta}}}\left[ \sum_{t\geq 0}\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) A_{\pi_{\theta}}(s_t,a_t) \right]:=g^{\gamma},$$

that is we drop the discount factor $$\gamma^t$$. This relates to the policy gradient theorem and that we want to sample states from the steady-state distribution instead of the discounted state distribution.

Now, we want to approximate $$A_{\pi}(s_t,a_t)$$ by an estimator $$\hat{A_t}$$. Schulman et al in the GAE paper introduce the definition of a $$\gamma$$-just estimator. The exact definition is maybe not essential, but $$\gamma$$-just estimators include $$Q_{\pi}(s_t,a_t)$$, $$A_{\pi}(s_t,a_t)$$ and the GAE estimator used in the PPO paper.

So, if $$\hat{A}_t$$ is $$\gamma$$-just, then we have: $$g^{\gamma} = \mathbb{E}_{\tau\sim p_{\pi_{\theta}}}\left[ \sum_{t\geq 0}\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) \hat{A}_t \right].$$

This expectation is however still not convenient for a practical algorithm, as we need to collect the trajectories with a policy with older weights $$\pi_{\theta_{\text{old}}}$$. So we approximate

$$g^{\gamma} \approx \mathbb{E}_{\tau\sim p_{\pi_{\theta_{\text{old}}}}}\left[ \sum_{t\geq 0}\nabla_{\theta}\log\pi_{\theta}(a_t\mid s_t) \hat{A}_t \right]:=\hat{g}.$$

Now, we need to approximate this expectation, so we collect $$N$$ rollouts of identical length $$T$$ (for simplicity) with the policy $$\pi_{\theta_{\text{old}}}$$, then we have an approximation of $$g^{\gamma}$$:

$$\nabla_{\theta} J_2(\theta) \approx g^{\gamma}\approx \hat{g}\approx\frac{1}{N}\sum_{n=0}^N\sum_{t=0}^T \nabla_{\theta}\log\pi_{\theta}(a_t^n\mid s_t^n) \hat{A}_t^n,$$ which is what the $$\mathbb{E}_t$$ in the PPO paper means. The expression on the right-hand side is now finally useful for gradient descent-like algorithms, as $$\hat{A_t}$$does not depend on $$\theta$$.

Note also that much of the PPO paper aims to ensure that the last approximation by replacing $$p_{\pi_{\theta}}$$ by $$p_{\pi_{\theta_{\text{old}}}}$$ is valid.

• Thanks for the detailed explanation. Indeed, it does seem like the confusion stems from the $\hat{E}_{t}$ notation. I understand your derivation but I dont get why your last estimator $\hat{g}$ is what they refer to by $\hat{E}_{t}$. For me "an empirical batch over a finite batch of samples" is rather your last estimor without the sum over t, i.e. $\frac{1}{N} \sum_{n = 0}^{N} \nabla_{\theta} \log \pi_{\theta}(a_{t}^{n} \mid s_{t}^{n}) \hat{A}_{t}^{n}$. Jul 8, 2022 at 16:25
• We want to estimate $\hat{g}$, which is an expectation over trajectories. Depending on the algorithm prior to PPO, this is done differently. It relates to the discussion of TD(0) vs n-step TD estimation (although the quantity in the expectation is different, the principle is similar). What you propose in the comment is to take $n$ samples of 1-step tranistions, which is often done in off-policy methods (like DDPG). The other way, done in VPG is to collect one trajectory of length $T$. You can also take $n$ trajectories of length $T$ as in A3C. The A3C paper shows many ways to estimate it. Jul 9, 2022 at 18:08