On-Policy Algorithms like PPO directly maximize the performance objective or an approximation of it. They tend to be quite stable and reliable but are often sample inefficient. Off-Policy Algorithms like TD3 improve the sample inefficiency by reusing data collected with previous policies, but they tend to be less stable. (Source: Kinds of RL Algorithms - Spinning up - OpenAI)

Looking at learning curves comparing SOTA algorithms, we see that off-policy algorithms quickly improve performance at the training's beginning. Here an example: 3M timestep benchmark of different algorithms for Walker2d-v3

Can we start training off-policy and after some time use the learned and quickly improved policy to init the policy network of an on-policy algorithm?


In the DRL nanodegree in Udacity, the instructor says it is possible to combine on- and off-policy learning and suggests the following paper where this has been done: Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic (ICLR 2017). Citing the paper:

The core idea is to use the first-order Taylor expansion of the critic as a control variate, resulting in an analytical gradient term through the critic and a Monte Carlo policy gradient term consisting of the residuals in advantage approximations. The method helps unify policy gradient and actor-critic methods: it can be seen as using the off-policy critic to reduce variance in policy gradient or using on-policy Monte Carlo returns to correct for bias in the critic gradient.

The authors provide an open source implementation of it in https://github.com/shaneshixiang/rllabplusplus

There is a follow-up paper by the same authors also addressing this problem: Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep Reinforcement Learning (NIPS 2017).

The Related Work section in both papers might be also worth looking at.

  • $\begingroup$ Dear @user5093249, thank you very much for your detailed and definitely interesting answer. I'll definitely have a look at the papers. I'm especially interested in combining existing algorithms (like TD3 and PPO) instead of developing new ones that combine on- and off-policy approaches. $\endgroup$ – Ray Walker May 14 '20 at 18:59
  • $\begingroup$ @RayWalker your approach seems to be in line with what the authors suggest. In fact, in the first paper, they state that "since Q-Prop uses both on-policy policy updates and off-policy critic learning, it can take advantage of prior work along both lines of research". They implemented it on top of TRPO-GAE for example. $\endgroup$ – user5093249 May 14 '20 at 19:56
  • $\begingroup$ Ok great, thank you very much for the additional information. I've now had a look at both abstracts and accepted your answer. It very well answers the main question in the topic. And the Q-Prop paper just jumped to the top of my reading list : ) $\endgroup$ – Ray Walker May 15 '20 at 18:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.