0
$\begingroup$

I understand that TRPO is a on-policy RL method and that it optimizes an expectation of the advantage or accumulated returns function over actions taken according to policy $\pi$.

Is it possible to use a replay buffer where the policy can be trained with returns gathered with old policies?

Would importance sampling and the KL divergence constraint prevent the policy from making bad updates?

Also, in PPO, it is stated in the algorithm that the policy is trained with the same trajectories over multiple iterations. Is this applicable for TRPO as well, seeing both algorithms are relatively similar?

Improve this question
$\endgroup$
1
  • $\begingroup$ It seems you're asking multiple distinct, although somehow related, questions here. I would recommend that you focus on one question in this post and remove the others, and move the others to their own separate post. $\endgroup$
    – nbro
    Commented Jan 21, 2023 at 17:58

1 Answer 1

Reset to default
-1
$\begingroup$

Is it possible to use a replay buffer with TRPO?

YES

You can check out the ACER algorithm (paper | code)

Given the formula for the policy gradient: $$ \nabla J(\theta) = \mathbb{E}_{a,s \sim \pi_\theta} \Big[ \nabla \log \pi_\theta (a|s) R(s,a) \Big], $$ there are two questions that need to be answered:

  1. How to compute the return $R(s,t)$ ?
  2. How to update the policy weights with this gradient ?

TRPO is an algorithm that deals with the second question.
Instead of updating the weights in the direction of the gradient (vanilla PG),TRPO suggests updating the weights using the natural gradient.
Regarding the first question there are a few options:

  • Monte-Carlo estimate: $R(s_t, a_t) = \sum_{i=t}^{T} r_{i+1}$;
  • one-step bootstrap using a value network: $R(s_t, a_t) = r_{t+1} + V_\phi (s_{t+1})$;
  • using q-value network: $R(s_t, a_t) = Q_\psi(s_t, a_t) $.

In case you decide to use a $Q$-function, then you can fit that function using standard techniques from Q-learning, e.g. replay buffer, double learning and so on. You can also check this out: https://youtu.be/7C2DSdXX-kQ?t=449&si=xl1nbYUMyE-F_QWU

Can you train with TRPO for multiple iterations just like PPO?

NO

PPO and TRPO are two different algorithms that try to achieve the same goal. Given the current data that you have, what is the biggest possible update step you can take on the policy parameters. TRPO follows the natural gradient and dynamically adjust the learning rate by solving a second order equation. PPO applies multiple update iterations and clips the updates if they are too large.

If you want to read more about PPO, feel free to check this blog post that I wrote:
https://pi-tau.github.io/posts/actor-critic/#ppo

Improve this answer
$\endgroup$

You must log in to answer this question.