We’re rewarding the question askers & reputations are being recalculated! Read more.

Questions tagged [off-policy]

For questions related to off-policy reinforcement learning algorithms, which estimate a policy (the target policy) while using another policy (the behavior policy), during the learning process, which ensures that all states are sufficiently explored. An example of an off-policy algorithm is Q-learning.

Filter by
Sorted by
Tagged with
0
votes
0answers
13 views

How is the general return-based off-policy equation derived?

I'm wondering how is the general return-based off-policy equation in Safe and efficient off-policy reinforcement learning derived $$\mathcal{R} Q(x, a):=Q(x, a)+\mathbb{E}_{\mu}\left[\sum_{t \geq 0} \...
3
votes
1answer
57 views

What is a non-starving policy in reinforcement learning?

In the paper, Eligibility Traces for off-Policy Policy Evaluation (2010), by Doina Precup et al., mentioned the term "non-starving" many times. The specific use of the term was like "non-starving ...
1
vote
1answer
30 views

Do I need to store the policy for RL?

I am creating a zero-sum game with RL and wondered if I need to store the policy, or if there are other RL methods that produce similar results (consistently beating the human player) without the need ...
3
votes
1answer
82 views

Understanding the n-step off-policy SARSA update

In Sutton & Barto's book (2nd ed) page 149, there is the equation 7.11 I am having a hard time understanding this equation. I would have thought that we should be moving $Q$ towards $G$, where $...
1
vote
1answer
48 views

On-policy distribution for Emphatic TD

The book by Sutton and Barto discussed in section 11.8 that the convergence of off-policy TD function approximation can be improved by correcting for distribution of states encountered. The section ...
1
vote
1answer
181 views

What is the relation between online learning and on-policy algorithms?

In the context of RL, there is the notion of on-policy and off-policy algorithms. I roughly understand the difference between on-policy and off-policy algorithms. Moreover, in RL, there's also the ...
5
votes
1answer
98 views

How do I compute the variance of the return of an evaluation policy using two behaviour policies?

Suppose there is an evaluation policy called $\pi_{e}$ and there are two behavior policies $\pi_{b1}$ and $\pi_{b2}$. I know that it is possible to estimate the return of policy $\pi_{e}$ through ...
1
vote
1answer
71 views

Why is the actor-critic algorithm limited to using on-policy data?

Why is the actor-critic algorithm limited to using on-policy data? Or can we use the actor-critic algorithm with off-policy data?
2
votes
2answers
428 views

What is the difference between on and off-policy deterministic actor-critic?

In the paper Deterministic Policy Gradient Algorithms, I am really confused about chapter 4.1 and 4.2 which is "On and off-policy Deterministic Actor-Critic". I don't know what's the difference ...