1
$\begingroup$

On-Policy Algorithms like PPO directly maximize the performance objective or an approximation of it. They tend to be quite stable and reliable but are often sample inefficient. Off-Policy Algorithms like TD3 improve the sample inefficiency by reusing data collected with previous policies, but they tend to be less stable. (Source: Kinds of RL Algorithms - Spinning up - OpenAI)

Looking at learning curves comparing SOTA algorithms, we see that off-policy algorithms quickly improve performance at the training's beginning. Here an example: 3M timestep benchmark of different algorithms for Walker2d-v3

Can we start training off-policy and after some time use the learned and quickly improved policy to init the policy network of an on-policy algorithm?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

In the DRL nanodegree in Udacity, the instructor says it is possible to combine on- and off-policy learning and suggests the following paper where this has been done: Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic (ICLR 2017). Citing the paper:

The core idea is to use the first-order Taylor expansion of the critic as a control variate, resulting in an analytical gradient term through the critic and a Monte Carlo policy gradient term consisting of the residuals in advantage approximations. The method helps unify policy gradient and actor-critic methods: it can be seen as using the off-policy critic to reduce variance in policy gradient or using on-policy Monte Carlo returns to correct for bias in the critic gradient.

The authors provide an open source implementation of it in https://github.com/shaneshixiang/rllabplusplus

There is a follow-up paper by the same authors also addressing this problem: Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning (NIPS 2017).

The Related Work section in both papers might be also worth looking at.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Dear @user5093249, thank you very much for your detailed and definitely interesting answer. I'll definitely have a look at the papers. I'm especially interested in combining existing algorithms (like TD3 and PPO) instead of developing new ones that combine on- and off-policy approaches. $\endgroup$
    – Ray Walker
    May 14, 2020 at 18:59
  • $\begingroup$ @RayWalker your approach seems to be in line with what the authors suggest. In fact, in the first paper, they state that "since Q-Prop uses both on-policy policy updates and off-policy critic learning, it can take advantage of prior work along both lines of research". They implemented it on top of TRPO-GAE for example. $\endgroup$
    – user5093249
    May 14, 2020 at 19:56
  • $\begingroup$ Ok great, thank you very much for the additional information. I've now had a look at both abstracts and accepted your answer. It very well answers the main question in the topic. And the Q-Prop paper just jumped to the top of my reading list : ) $\endgroup$
    – Ray Walker
    May 15, 2020 at 18:40

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .