0
$\begingroup$

Vanilla policy gradient has a loss function:

$\mathcal{L}_{\pi_{\theta}(\theta)} = E_{\tau \sim \pi_{\theta}}[\sum\limits_{t = 0}^{\infty}\gamma^{t}r_{t}]$

while in TRPO it is:

$\mathcal{L}_{\pi_{\theta_{old}}(\theta)} = \frac{1}{1 - \gamma}E_{s, a \sim \pi_{\theta_{old}}}[\frac{\pi_{\theta}(a|s)}{\pi_{\theta_{old}(a|s)}}A^{\pi_{\theta_{old}}}(s,a)]$

There exists some problems of vanilla policy gradient, such that distance dismatch between parameter space and policy space and suffering poor sample efficiency.

For tackling this problem, where in TRPO, it introduces importance sampling for improving this. However when I compared the pseudocode of two algorithms, I don't see any obvious evidence that aids the point. They all firstly sampled multiple trajectories under current policy, wherein former one then just using the estimated gradient of loss function to update the parameter $\theta$, whereas in the latter one, also using the estimated gradient but under KL constraint to update the parameter $\theta$. It looks like they are almost in the same processure but with some subtle difference of the optimization progress.

And my question is that should it be more accurate to say that importance sampling just increases the stability of the sequential decision making process, or rather, avoid update too aggressively, but not directly improves the sample efficiency?

Is there any suggestion on this problem? Many thanks!

enter image description here

enter image description here

Given pseudocode of two algorithms, source from openai spinning up document.

$\endgroup$
2
  • $\begingroup$ You say "when I compared the pseudocode of two algorithms, I don't see any obvious evidence that aids the point" - It might be a good idea to specify which pseudocode you're reading, because, I guess, there might be multiple pseudocodes circulating, especially of vanilla PG. I would also recommend that you use the tags policy-gradients and trpo. $\endgroup$
    – nbro
    Feb 16 at 21:06
  • $\begingroup$ @nbro Thanks, revised! $\endgroup$ Feb 17 at 6:15

1 Answer 1

0
$\begingroup$

The point of importance sampling is to use the same episode(s) to do multiple policy gradient updates. It will definitely increase sample efficiency over the strictly on-policy case, simply because in an off-policy algorithm we can use a single transition to do multiple updates, whereas we only get a single update from that same experience in the on-policy case.

First of all, in the psuedo-code for TRPO that you give, I see no importance sampling being done, and the psuedo-code makes it seem that the algorithm is being done on-policy.

Basically, an on policy algorithm would look like: \begin{align*} & \text{ for each iteration }: \\ & \qquad \text{ for t in range(n)}: \\ & \qquad \qquad \text{sample } a_t; \text{ get reward } r_t \text{ and next state } s_{t+1} \\ & \qquad \text{ do policy update based on } a_1, r_1, s_2, a_2, \ldots \end{align*} whereas an off policy algorithm is like \begin{align*} & \text{ for each iteration }: \\ & \qquad \text{ for t in range(n)}: \\ & \qquad \qquad \text{sample } a_t; \text{ get reward } r_t \text{ and next state } s_{t+1} \\ & \qquad \text{ for m epochs}: \\ & \qquad \qquad \text{ for k mini-batches}: \\ & \qquad \qquad \qquad \text{make mini batch from} \{a_1, r_1, s_2, a_2, \ldots\} \text{ and do policy gradient update } \end{align*} So in an off-policy algorithm you may have something like $m=10$ (10 epochs), meaning every transition you do gets used in 10 policy gradient updates. An on policy algorithm would only use that transition once.

$\endgroup$
3
  • $\begingroup$ This clarifies a lot. But I still have some problems. 1) TRPO is a on-policy method for sure, for which it confuses me a lot that how is sample efficiency attained without doing multiple update like the off-policy method does (seems no essential difference bewteen VPG). 2) For best of my knowledge, is policy gradient method always on-policy because the object of both sampling and control is $\pi_{\theta}$ under the same parameter space? So if I'm not wrong, how to do mini-batches without introducing a behavior policy? $\endgroup$ Feb 18 at 6:38
  • $\begingroup$ And I also wanna stress that you've concluded at the beginning with the statement It will definitely increase sample efficiency over the strictly on-policy case, however, your explanation right after mainly focuses on the off-policy case where can attain sample efficiency by runing many epoches, but not directly aids the conclusion of on-policy case. $\endgroup$ Feb 18 at 6:43
  • $\begingroup$ my answer explains what is meant by 'off policy' in the context of the PPO algorithm (which is mentioned in your title, and is essentially the successor to TRPO). I'm not exactly sure what purpose importance sampling serves in TRPO. $\endgroup$
    – Taw
    Feb 19 at 23:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .