17 votes

What is the relation between online (or offline) learning and on-policy (or off-policy) algorithms?

The concepts of on-policy vs off-policy and online vs offline are separate, but do interact to make certain combinations more feasible. When looking at this, it is worth also considering the ...
user avatar
  • 23.1k
7 votes

Why do we need importance sampling?

Importance sampling is typically used when the distribution of interest is difficult to sample from - e.g. it could be computationally expensive to draw samples from the distribution - or when the ...
user avatar
5 votes
Accepted

How can the importance sampling ratio be different than zero when the target policy is deterministic?

You're correct, when the target policy $\pi$ is deterministic, the importance sampling ratio will be $\geq 1$ along the trajectory where the behaviour policy $b$ happened to have taken the same ...
user avatar
  • 9,316
5 votes
Accepted

Why is the n-step tree backup algorithm an off-policy algorithm?

As for your first two questions: there is indeed a behaviour and a target policy, which can be different. In the example image of the $3$-step tree-backup update in the beginning of the section you ...
user avatar
  • 9,316
5 votes

Why are Q values updated according to the greedy policy?

Short answer The Q values are updated using a greedy policy because, in the Q-learning algorithm, the $\max$ operator is used to determine the target, which is denoted by $$\color{green}{R_{t+1}} + \...
user avatar
  • 33k
5 votes

Do off-policy policy gradient methods exist?

Absolutely, it’s a really interesting problem. Here is a paper detailing off policy actor critic. This is important because this method can also support continuous actions. The general idea of off-...
user avatar
5 votes
Accepted

Why can we take the action $a$ from the next state $s'$ in the max part of the Q-learning update rule, if that action doesn't lead to any reward?

I'm using OpenAI's cartpole environment. First of all, is this environment not Markov? The OpenAI Gym CartPole environment is Markov. Whether or not you know the transition probabilities does not ...
user avatar
  • 23.1k
4 votes
Accepted

What is the difference between on and off-policy deterministic actor-critic?

The twist here is that the $a_{t+1}$ in (11) and the $\mu(s_{t+1})$ in (16) are the same and actually the $a_t$ in the on-policy case and the $a_t$ in the off-policy case are different. The key to ...
user avatar
4 votes
Accepted

What is a non-starving policy in reinforcement learning?

A non-starving policy is a (behavior) policy that is theoretically guaranteed to visit each state and take all possible actions from each state an infinite number of times, so that to always update $Q(...
user avatar
  • 33k
4 votes

What is the intuition behind importance sampling for off-policy value evaluation?

Recall that our goal is to be able to accurately estimate the true value of each state by computing a sample average over returns starting from that state: $$v_{q}(s) \doteq \mathbb{E}_{q}\left[G_{t} |...
user avatar
4 votes

Why does off-policy learning outperform on-policy learning?

This post contains many answers that describe the difference between on-policy vs. off-policy. Your book may be referring to how the current (DQN-based) state-of-the-art (SOTA) algorithms, such as Ape-...
user avatar
  • 131
4 votes
Accepted

Which policy do I need to use in updating Q function?

I am going to stick with Q learning here to keep things simple. Most value-based reinforcement learning used for optimal control will have some statement similar to: Choose $a$ from $s$ using policy ...
user avatar
  • 23.1k
3 votes

Why is DDPG an off-policy RL algorithm?

DDPG is an off-policy algorithm simply because of the objective taking expectation with respect to some other distribution that we are not learning about, i.e. the deterministic policy gradient can be ...
user avatar
3 votes
Accepted

What is meant by "generate the data" in describing the difference between on-policy and off-policy?

In the book, the phrase "generate the data" refers to the data from observations about states, actions, next states and rewards, that then get used to make value estimate updates. In both ...
user avatar
  • 23.1k
3 votes
Accepted

Is Expected SARSA an off-policy or on-policy algorithm?

Expected SARSA can be used either on-policy or off-policy. The policy that you use in the update step determines which it is. If the update step uses a different weighting for action choices than the ...
user avatar
  • 23.1k
3 votes
Accepted

Are model-free and off-policy algorithms the same?

In respect of RL, is model-free and off-policy the same thing, just different terminology? No, they are entirely different terms, with the only thing they have in common is that they are both ways ...
user avatar
  • 23.1k
3 votes
Accepted

Understanding the n-step off-policy SARSA update

Multiplying the entire update by $\rho$ has the desirable property that experience affects $Q$ less when the behavior policy is unrelated to the target policy. In the extreme, if the trajectory taken ...
user avatar
3 votes
Accepted

Can I add expert data to the replay buffer used by the DDPG algorithm in order to make it converge faster?

What I want to know is whether I can add expert data to the replay buffer, given that DDPG is an off-policy algorithm? You certainly can, that is indeed one of the advantages of off-policy learning ...
user avatar
  • 9,316
3 votes

With Monte Carlo off-policy learning what do we correct by using importance sampling?

We estimate a value using sampling on whole episodes, and we take this values to construct the target policy. The crucial bit that you are missing is that there is no single value of $V(s)$ (or $Q(s,...
user avatar
  • 1,763
3 votes
Accepted

Which policy has to be followed by a player while construction of its own Q-table?

I'll assume Q-player is being trained with Q learning (note, Q tables can be useful in other algorithms too, like SARSA). Q learning is an off policy algorithm, meaning that the Q values can be ...
user avatar
  • 961
2 votes

Why is $M_t$ (the emphasis) helping in correcting for the state distribution in the Emphatic TD algorithm?

I don't think the section was written in haste. I think they just didn't have space to include the whole proof. It's a bit involved, so they just gave concepts. An Emphatic Approach to the Problem ...
user avatar
2 votes
Accepted

Understanding the W term in off policy monte carlo learning

The pseudocode you have copied looks incorrect to me, and I think it is from the first edition. The main issue is at the end of the loop. Where the book has $\qquad W \leftarrow W \frac{1}{\mu(A_t|...
user avatar
  • 23.1k
2 votes
Accepted

Could we update the policy network with previous trajectories using supervised learning?

You cannot really do that because you have no way of knowing how good the action really is to make reasonable labels for supervised learning (that's the whole point why we need reinforcement learning)....
user avatar
  • 2,216
2 votes

What is the intuition behind importance sampling for off-policy value evaluation?

In the application of importance sampling to RL, is the expectation of the function $f$ equivalent to the value of the trajectories, which is represented by the trajectories $x$? I believe what you ...
user avatar
2 votes
Accepted

What is the difference between on-policy and off-policy for continuous environments?

First, some preliminary questions: in this case, what is the optimal policy? It is the policy that maximises return from any given time step $G_t$. You need to be careful with your definition of ...
user avatar
  • 23.1k
2 votes

What is the intuition behind importance sampling for off-policy value evaluation?

Let's fix some notation: we're collecting data from behavior policy $\pi_0$ and we want to evaluate a policy $\pi$. Of course, if we had plenty of data from policy $\pi$ that would be the best way to ...
user avatar
  • 131
2 votes
Accepted

How does this TD(0) off-policy value update formula work?

This would mean we decrease the value of this state. Yes. This update that reduces the estimate is correct because it adjusts for the inevitable over-estimate of value when the exploration policy ...
user avatar
  • 23.1k
1 vote

When learning off-policy with multi-step returns, why do we use the current behaviour policy in importance sampling?

According to my understanding, you don't use just the current behavior policy for sampling. The importance sampling ratio is calculated as the product of the probability ratios for both the target and ...
user avatar
1 vote

Learning only using off-policy samples

What you're describing is off-policy learning. A classic example is $Q$-learning, where you follow some policy $\pi$ whilst learning about the greedy policy. If you're interested in actor-critic ...
user avatar
1 vote

How do I know that the DQN has learnt an appropriate Q function?

DQN is famous for doing over-approximation on Q function. However, having over approximated Q does not imply that it does not perform well in the environment. (unless it looks ridiculously high) From ...
user avatar
  • 11

Only top scored, non community-wiki answers of a minimum length are eligible