# Does importance sampling really improve sampling efficiency of policy gradient methods such as TRPO or PPO?

Vanilla policy gradient has a loss function:

$$\mathcal{L}_{\pi_{\theta}(\theta)} = E_{\tau \sim \pi_{\theta}}[\sum\limits_{t = 0}^{\infty}\gamma^{t}r_{t}]$$

while in TRPO it is:

$$\mathcal{L}_{\pi_{\theta_{old}}(\theta)} = \frac{1}{1 - \gamma}E_{s, a \sim \pi_{\theta_{old}}}[\frac{\pi_{\theta}(a|s)}{\pi_{\theta_{old}(a|s)}}A^{\pi_{\theta_{old}}}(s,a)]$$

There exists some problems of vanilla policy gradient, such that distance dismatch between parameter space and policy space and suffering poor sample efficiency.

For tackling this problem, where in TRPO, it introduces importance sampling for improving this. However when I compared the pseudocode of two algorithms, I don't see any obvious evidence that aids the point. They all firstly sampled multiple trajectories under current policy, wherein former one then just using the estimated gradient of loss function to update the parameter $$\theta$$, whereas in the latter one, also using the estimated gradient but under KL constraint to update the parameter $$\theta$$. It looks like they are almost in the same processure but with some subtle difference of the optimization progress.

And my question is that should it be more accurate to say that importance sampling just increases the stability of the sequential decision making process, or rather, avoid update too aggressively, but not directly improves the sample efficiency?

Is there any suggestion on this problem? Many thanks!

Given pseudocode of two algorithms, source from openai spinning up document.

• You say "when I compared the pseudocode of two algorithms, I don't see any obvious evidence that aids the point" - It might be a good idea to specify which pseudocode you're reading, because, I guess, there might be multiple pseudocodes circulating, especially of vanilla PG. I would also recommend that you use the tags policy-gradients and trpo.
– nbro
Feb 16 at 21:06
• @nbro Thanks, revised! Feb 17 at 6:15

Basically, an on policy algorithm would look like: \begin{align*} & \text{ for each iteration }: \\ & \qquad \text{ for t in range(n)}: \\ & \qquad \qquad \text{sample } a_t; \text{ get reward } r_t \text{ and next state } s_{t+1} \\ & \qquad \text{ do policy update based on } a_1, r_1, s_2, a_2, \ldots \end{align*} whereas an off policy algorithm is like \begin{align*} & \text{ for each iteration }: \\ & \qquad \text{ for t in range(n)}: \\ & \qquad \qquad \text{sample } a_t; \text{ get reward } r_t \text{ and next state } s_{t+1} \\ & \qquad \text{ for m epochs}: \\ & \qquad \qquad \text{ for k mini-batches}: \\ & \qquad \qquad \qquad \text{make mini batch from} \{a_1, r_1, s_2, a_2, \ldots\} \text{ and do policy gradient update } \end{align*} So in an off-policy algorithm you may have something like $$m=10$$ (10 epochs), meaning every transition you do gets used in 10 policy gradient updates. An on policy algorithm would only use that transition once.
• This clarifies a lot. But I still have some problems. 1) TRPO is a on-policy method for sure, for which it confuses me a lot that how is sample efficiency attained without doing multiple update like the off-policy method does (seems no essential difference bewteen VPG). 2) For best of my knowledge, is policy gradient method always on-policy because the object of both sampling and control is $\pi_{\theta}$ under the same parameter space? So if I'm not wrong, how to do mini-batches without introducing a behavior policy? Feb 18 at 6:38