20
votes
What is the relation between online (or offline) learning and on-policy (or off-policy) algorithms?
The concepts of on-policy vs off-policy and online vs offline are separate, but do interact to make certain combinations more feasible. When looking at this, it is worth also considering the ...
- 30.3k
7
votes
Why do we need importance sampling?
Importance sampling is typically used when the distribution of interest is difficult to sample from - e.g. it could be computationally expensive to draw samples from the distribution - or when the ...
- 4,665
6
votes
Do off-policy policy gradient methods exist?
Absolutely, it’s a really interesting problem. Here is a paper detailing off policy actor critic. This is important because this method can also support continuous actions.
The general idea of off-...
- 3,787
6
votes
Accepted
Why is the n-step tree backup algorithm an off-policy algorithm?
As for your first two questions: there is indeed a behaviour and a target policy, which can be different. In the example image of the $3$-step tree-backup update in the beginning of the section you ...
- 10.1k
5
votes
Why are Q values updated according to the greedy policy?
Short answer
The Q values are updated using a greedy policy because, in the Q-learning algorithm, the $\max$ operator is used to determine the target, which is denoted by
$$\color{green}{R_{t+1}} + \...
- 39.6k
5
votes
Why does off-policy learning outperform on-policy learning?
This post contains many answers that describe the difference between on-policy vs. off-policy.
Your book may be referring to how the current (DQN-based) state-of-the-art (SOTA) algorithms, such as Ape-...
- 151
5
votes
Accepted
Why can we take the action $a$ from the next state $s'$ in the max part of the Q-learning update rule, if that action doesn't lead to any reward?
I'm using OpenAI's cartpole environment. First of all, is this environment not Markov?
The OpenAI Gym CartPole environment is Markov. Whether or not you know the transition probabilities does not ...
- 30.3k
4
votes
Accepted
What is the difference between on and off-policy deterministic actor-critic?
The twist here is that the $a_{t+1}$ in (11) and the $\mu(s_{t+1})$ in (16) are the same and actually the $a_t$ in the on-policy case and the $a_t$ in the off-policy case are different.
The key to ...
- 552
4
votes
Accepted
How can the importance sampling ratio be different than zero when the target policy is deterministic?
You're correct, when the target policy $\pi$ is deterministic, the importance sampling ratio will be $\geq 1$ along the trajectory where the behaviour policy $b$ happened to have taken the same ...
- 10.1k
4
votes
Accepted
Are model-free and off-policy algorithms the same?
In respect of RL, is model-free and off-policy the same thing, just different terminology?
No, they are entirely different terms, with the only thing they have in common is that they are both ways ...
- 30.3k
4
votes
Accepted
What is a non-starving policy in reinforcement learning?
A non-starving policy is a (behavior) policy that is theoretically guaranteed to visit each state and take all possible actions from each state an infinite number of times, so that to always update $Q(...
- 39.6k
4
votes
What is the intuition behind importance sampling for off-policy value evaluation?
Recall that our goal is to be able to accurately estimate the true value of each state by computing a sample average over returns starting from that state:
$$v_{q}(s) \doteq \mathbb{E}_{q}\left[G_{t} |...
- 722
4
votes
Accepted
Which policy do I need to use in updating Q function?
I am going to stick with Q learning here to keep things simple. Most value-based reinforcement learning used for optimal control will have some statement similar to:
Choose $a$ from $s$ using policy ...
- 30.3k
3
votes
Accepted
What is meant by "generate the data" in describing the difference between on-policy and off-policy?
In the book, the phrase "generate the data" refers to the data from observations about states, actions, next states and rewards, that then get used to make value estimate updates.
In both ...
- 30.3k
3
votes
Why is DDPG an off-policy RL algorithm?
DDPG is an off-policy algorithm simply because of the objective taking expectation with respect to some other distribution that we are not learning about, i.e. the deterministic policy gradient can be ...
- 4,665
3
votes
Accepted
Is Expected SARSA an off-policy or on-policy algorithm?
Expected SARSA can be used either on-policy or off-policy.
The policy that you use in the update step determines which it is. If the update step uses a different weighting for action choices than the ...
- 30.3k
3
votes
Accepted
Understanding the n-step off-policy SARSA update
Multiplying the entire update by $\rho$ has the desirable property that experience affects $Q$ less when the behavior policy is unrelated to the target policy. In the extreme, if the trajectory taken ...
- 2,028
3
votes
Accepted
Can I add expert data to the replay buffer used by the DDPG algorithm in order to make it converge faster?
What I want to know is whether I can add expert data to the replay buffer, given that DDPG is an off-policy algorithm?
You certainly can, that is indeed one of the advantages of off-policy learning ...
- 10.1k
3
votes
With Monte Carlo off-policy learning what do we correct by using importance sampling?
We estimate a value using sampling on whole episodes, and we take this values to construct the target policy.
The crucial bit that you are missing is that there is no single value of $V(s)$ (or $Q(s,...
- 2,426
3
votes
Accepted
Which policy has to be followed by a player while construction of its own Q-table?
I'll assume Q-player is being trained with Q learning (note, Q tables can be useful in other algorithms too, like SARSA).
Q learning is an off policy algorithm, meaning that the Q values can be ...
- 1,126
2
votes
Why is $M_t$ (the emphasis) helping in correcting for the state distribution in the Emphatic TD algorithm?
I don't think the section was written in haste. I think they just didn't have space to include the whole proof. It's a bit involved, so they just gave concepts.
An Emphatic Approach to the Problem
...
- 2,028
2
votes
Accepted
Could we update the policy network with previous trajectories using supervised learning?
You cannot really do that because you have no way of knowing how good the action really is to make reasonable labels for supervised learning (that's the whole point why we need reinforcement learning)....
- 2,336
2
votes
What is the intuition behind importance sampling for off-policy value evaluation?
In the application of importance sampling to RL, is the expectation of the function $f$ equivalent to the value of the trajectories, which is represented by the trajectories $x$?
I believe what you ...
- 4,665
2
votes
Accepted
What is the difference between on-policy and off-policy for continuous environments?
First, some preliminary questions: in this case, what is the optimal policy?
It is the policy that maximises return from any given time step $G_t$. You need to be careful with your definition of ...
- 30.3k
2
votes
What is the intuition behind importance sampling for off-policy value evaluation?
Let's fix some notation: we're collecting data from behavior policy $\pi_0$ and we want to evaluate a policy $\pi$.
Of course, if we had plenty of data from policy $\pi$ that would be the best way to ...
- 151
2
votes
Accepted
Understanding the W term in off policy monte carlo learning
The pseudocode you have copied looks incorrect to me, and I think it is from the first edition.
The main issue is at the end of the loop. Where the book has
$\qquad W \leftarrow W \frac{1}{\mu(A_t|...
- 30.3k
2
votes
Accepted
How does this TD(0) off-policy value update formula work?
This would mean we decrease the value of this state.
Yes. This update that reduces the estimate is correct because it adjusts for the inevitable over-estimate of value when the exploration policy ...
- 30.3k
2
votes
Accepted
Can off-policy algorithms benefit from the parallelization?
From the point of view of someone developing an in-house DRL lib and working on extremely CPU-intensive environments (usually large finite element-based simulations that can require several hours to ...
- 136
2
votes
Accepted
Why does HER not work with on-policy RL algorithms?
On policy algorithms contain policy and/or value update calculations that assume data was generated by the current policy. Breaking that assumption will cause them to miscalculate, or not function at ...
- 30.3k
2
votes
Why does HER not work with on-policy RL algorithms?
In principle replay mechanisms as HER, PER, etc cannot be applied to on-policy algorithms, like SARSA and Policy gradient as stated by @Neil Slater.
Anyway, If you are willing to "adapt" ...
- 2,443
Only top scored, non community-wiki answers of a minimum length are eligible