Skip to main content
21 votes

What is the relation between online (or offline) learning and on-policy (or off-policy) algorithms?

The concepts of on-policy vs off-policy and online vs offline are separate, but do interact to make certain combinations more feasible. When looking at this, it is worth also considering the ...
Neil Slater's user avatar
  • 32.7k
7 votes

Why do we need importance sampling?

Importance sampling is typically used when the distribution of interest is difficult to sample from - e.g. it could be computationally expensive to draw samples from the distribution - or when the ...
David's user avatar
  • 4,920
6 votes

Do off-policy policy gradient methods exist?

Absolutely, it’s a really interesting problem. Here is a paper detailing off policy actor critic. This is important because this method can also support continuous actions. The general idea of off-...
Jaden Travnik's user avatar
6 votes
Accepted

Why is the n-step tree backup algorithm an off-policy algorithm?

As for your first two questions: there is indeed a behaviour and a target policy, which can be different. In the example image of the $3$-step tree-backup update in the beginning of the section you ...
Dennis Soemers's user avatar
  • 10.3k
5 votes

Why are Q values updated according to the greedy policy?

Short answer The Q values are updated using a greedy policy because, in the Q-learning algorithm, the $\max$ operator is used to determine the target, which is denoted by $$\color{green}{R_{t+1}} + \...
nbro's user avatar
  • 40.9k
5 votes

Why does off-policy learning outperform on-policy learning?

This post contains many answers that describe the difference between on-policy vs. off-policy. Your book may be referring to how the current (DQN-based) state-of-the-art (SOTA) algorithms, such as Ape-...
kaiwenw's user avatar
  • 151
5 votes
Accepted

Why can we take the action $a$ from the next state $s'$ in the max part of the Q-learning update rule, if that action doesn't lead to any reward?

I'm using OpenAI's cartpole environment. First of all, is this environment not Markov? The OpenAI Gym CartPole environment is Markov. Whether or not you know the transition probabilities does not ...
Neil Slater's user avatar
  • 32.7k
4 votes
Accepted

What is the difference between on and off-policy deterministic actor-critic?

The twist here is that the $a_{t+1}$ in (11) and the $\mu(s_{t+1})$ in (16) are the same and actually the $a_t$ in the on-policy case and the $a_t$ in the off-policy case are different. The key to ...
Hai Nguyen's user avatar
4 votes
Accepted

How can the importance sampling ratio be different than zero when the target policy is deterministic?

You're correct, when the target policy $\pi$ is deterministic, the importance sampling ratio will be $\geq 1$ along the trajectory where the behaviour policy $b$ happened to have taken the same ...
Dennis Soemers's user avatar
  • 10.3k
4 votes
Accepted

Are model-free and off-policy algorithms the same?

In respect of RL, is model-free and off-policy the same thing, just different terminology? No, they are entirely different terms, with the only thing they have in common is that they are both ways ...
Neil Slater's user avatar
  • 32.7k
4 votes
Accepted

What is a non-starving policy in reinforcement learning?

A non-starving policy is a (behavior) policy that is theoretically guaranteed to visit each state and take all possible actions from each state an infinite number of times, so that to always update $Q(...
nbro's user avatar
  • 40.9k
4 votes

What is the intuition behind importance sampling for off-policy value evaluation?

Recall that our goal is to be able to accurately estimate the true value of each state by computing a sample average over returns starting from that state: $$v_{q}(s) \doteq \mathbb{E}_{q}\left[G_{t} |...
user5093249's user avatar
4 votes
Accepted

Which policy do I need to use in updating Q function?

I am going to stick with Q learning here to keep things simple. Most value-based reinforcement learning used for optimal control will have some statement similar to: Choose $a$ from $s$ using policy ...
Neil Slater's user avatar
  • 32.7k
3 votes
Accepted

What is meant by "generate the data" in describing the difference between on-policy and off-policy?

In the book, the phrase "generate the data" refers to the data from observations about states, actions, next states and rewards, that then get used to make value estimate updates. In both ...
Neil Slater's user avatar
  • 32.7k
3 votes
Accepted

What is the difference between on-policy and off-policy for continuous environments?

First, some preliminary questions: in this case, what is the optimal policy? It is the policy that maximises return from any given time step $G_t$. You need to be careful with your definition of ...
Neil Slater's user avatar
  • 32.7k
3 votes

Why is DDPG an off-policy RL algorithm?

DDPG is an off-policy algorithm simply because of the objective taking expectation with respect to some other distribution that we are not learning about, i.e. the deterministic policy gradient can be ...
David's user avatar
  • 4,920
3 votes
Accepted

Is Expected SARSA an off-policy or on-policy algorithm?

Expected SARSA can be used either on-policy or off-policy. The policy that you use in the update step determines which it is. If the update step uses a different weighting for action choices than the ...
Neil Slater's user avatar
  • 32.7k
3 votes
Accepted

Can I add expert data to the replay buffer used by the DDPG algorithm in order to make it converge faster?

What I want to know is whether I can add expert data to the replay buffer, given that DDPG is an off-policy algorithm? You certainly can, that is indeed one of the advantages of off-policy learning ...
Dennis Soemers's user avatar
  • 10.3k
3 votes

With Monte Carlo off-policy learning what do we correct by using importance sampling?

We estimate a value using sampling on whole episodes, and we take this values to construct the target policy. The crucial bit that you are missing is that there is no single value of $V(s)$ (or $Q(s,...
Kostya's user avatar
  • 2,534
3 votes
Accepted

Which policy has to be followed by a player while construction of its own Q-table?

I'll assume Q-player is being trained with Q learning (note, Q tables can be useful in other algorithms too, like SARSA). Q learning is an off policy algorithm, meaning that the Q values can be ...
harwiltz's user avatar
  • 1,136
2 votes
Accepted

Understanding the n-step off-policy SARSA update

Multiplying the entire update by $\rho$ has the desirable property that experience affects $Q$ less when the behavior policy is unrelated to the target policy. In the extreme, if the trajectory taken ...
Philip Raeisghasem's user avatar
2 votes

Why is $M_t$ (the emphasis) helping in correcting for the state distribution in the Emphatic TD algorithm?

I don't think the section was written in haste. I think they just didn't have space to include the whole proof. It's a bit involved, so they just gave concepts. An Emphatic Approach to the Problem ...
Philip Raeisghasem's user avatar
2 votes
Accepted

Could we update the policy network with previous trajectories using supervised learning?

You cannot really do that because you have no way of knowing how good the action really is to make reasonable labels for supervised learning (that's the whole point why we need reinforcement learning)....
Brale's user avatar
  • 2,406
2 votes

What is the intuition behind importance sampling for off-policy value evaluation?

In the application of importance sampling to RL, is the expectation of the function $f$ equivalent to the value of the trajectories, which is represented by the trajectories $x$? I believe what you ...
David's user avatar
  • 4,920
2 votes

What is the intuition behind importance sampling for off-policy value evaluation?

Let's fix some notation: we're collecting data from behavior policy $\pi_0$ and we want to evaluate a policy $\pi$. Of course, if we had plenty of data from policy $\pi$ that would be the best way to ...
kaiwenw's user avatar
  • 151
2 votes
Accepted

Understanding the W term in off policy monte carlo learning

The pseudocode you have copied looks incorrect to me, and I think it is from the first edition. The main issue is at the end of the loop. Where the book has $\qquad W \leftarrow W \frac{1}{\mu(A_t|...
Neil Slater's user avatar
  • 32.7k
2 votes
Accepted

How does this TD(0) off-policy value update formula work?

This would mean we decrease the value of this state. Yes. This update that reduces the estimate is correct because it adjusts for the inevitable over-estimate of value when the exploration policy ...
Neil Slater's user avatar
  • 32.7k
2 votes
Accepted

Can off-policy algorithms benefit from the parallelization?

From the point of view of someone developing an in-house DRL lib and working on extremely CPU-intensive environments (usually large finite element-based simulations that can require several hours to ...
Scrimbibete's user avatar
2 votes
Accepted

How does off-policy Monte Carlo weighted importance sampling bias converge to zero (Sutton & Barto Section 5.5)

Short explanation The bias converges asymptotically to zero with more visits of the state $s$. The value function is estimated in the following way: \begin{equation} v_{\pi}(s) = \frac{\sum_{t \in \...
pythonic833's user avatar
2 votes
Accepted

Why does HER not work with on-policy RL algorithms?

On policy algorithms contain policy and/or value update calculations that assume data was generated by the current policy. Breaking that assumption will cause them to miscalculate, or not function at ...
Neil Slater's user avatar
  • 32.7k

Only top scored, non community-wiki answers of a minimum length are eligible