3
$\begingroup$

I am self-studying about Reinforcement Learning using different online resources. I now have a basic understanding of how RL works.

I saw this in a book:

Q-learning is an off-policy learner. An off-policy learner learns the value of an optimal policy independently of the agent’s actions, as long as it explores enough.

An on-policy learner learns the value of the policy being carried out by the agent, including the exploration steps.

However, I am not quite understanding the difference. Secondly, I came across that off-policy learner works better than the on-policy agent. I don't understand why that would be i.e. why off-policy would be better than the on-policy.

$\endgroup$
2
  • $\begingroup$ Here's a very related question stats.stackexchange.com/q/184657 $\endgroup$ – mugoh Nov 26 '20 at 4:42
  • 1
    $\begingroup$ You say "Secondly, I came across that off-policy learner works better than the on-policy agent. ". Where did you hear/read this? $\endgroup$ – nbro Nov 26 '20 at 9:49
4
$\begingroup$

This post contains many answers that describe the difference between on-policy vs. off-policy.

Your book may be referring to how the current (DQN-based) state-of-the-art (SOTA) algorithms, such as Ape-X, R2D2, Agent57 are technically "off-policy", since they use a (very large!) replay buffer, often filled in a distributed manner. This has a number of benefits, such as reusing experience and not forgetting important experiences.

Another benefit is that you can collect a lot of experience distributedly. Since RL is typically not bottlenecked by the computation for training but rather from collecting experiences, the distributed replay buffer in Ape-X can enable much faster training, in terms of seconds but not sample complexity.

However, it's important to emphasize that these replay-buffer approaches are almost on-policy, in the sense that the replay buffer is constantly updated with new experiences. So, the policy in the replay buffer is "not too different" from your current policy (just a few gradient steps away). Most importantly, this allows the policy to learn from its own mistakes if it makes any...

Off-policy learning, in general, can also refer to batch RL (a.k.a. offline RL), where you're provided a dataset of experiences from another behavior policy, and your goal is to improve over it. Notably, you don't get to rollout your current policy in any way! In this case, algorithms that worked well with a replay-buffer (like DQN, SAC) fail miserably, since they over-estimate the value of actions when they extrapolate outside the "support" of the dataset. See the BCQ paper which illustrates how a lot of "off-policy" algorithms like DQN fail when the "distance between the two policies is large". For this task, SOTA is a form of weighted behavioral cloning called Critic Regularized Regression (CRR).

It's also worth noting that importance sampling can correct off-policy gradients to be on-policy; but the farther away your target policy is, the larger the variance. This is especially deadly for long horizon tasks (often called curse of horizon).

To summarize, using replay-buffer (which makes the algorithm off-policy), especially a distributed one, can offer a lot of benefits over pure on-policy algorithms. However, this is a very special class of off-policy algorithms, where the behavioral policy is close to your policy.

But in general, off-policy is a lot harder than on-policy; you'll suffer from extrapolation bias if you use DQN-based approaches, and exponential variance blow-up if you use importance sampling to correct for it.

$\endgroup$
10
  • 1
    $\begingroup$ thanks for the response. But your answer goes way above my head :-( $\endgroup$ – Exploring Nov 26 '20 at 6:48
  • $\begingroup$ which part do you want clarification on? do you understand on-policy vs. off-policy after reading @mugoh's post? $\endgroup$ – kaiwenw Nov 26 '20 at 6:55
  • $\begingroup$ Maybe you should cite a paper that supports this "in this case, a lot of the traditional "off-policy" algos like DQN, SAC fail miserably, and the SOTA is a form of weighted behavioral cloning called Critic Regularized Regression (CRR)", for those not familiar with the SOTA. It's also not clear what you mean by the last sentence and in particular "but in general on-policy is much easier than off-policy!" (easier in terms of what?). Also, not sure why you say that DQN-based algorithms are "almost on-policy". That seems like saying that there are almost no off-policy algorithms. $\endgroup$ – nbro Nov 26 '20 at 10:42
  • $\begingroup$ I would actually say that off-policy algorithms can become on-policy, i.e. they generalize on-policy ones, but they are not "almost on-policy", if you want to be precise. Well, at least, I don't fully get what you mean by that. $\endgroup$ – nbro Nov 26 '20 at 10:46
  • 1
    $\begingroup$ This answer contains useful information, but I think it is pitched incorrectly to level of knowledge displayed in the question. $\endgroup$ – Neil Slater Nov 26 '20 at 12:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.