Questions tagged [off-policy-methods]

For questions related to off-policy reinforcement learning algorithms, which estimate a policy (the target policy) while using another policy (the behavior policy), during the learning process, which ensures that all states are sufficiently explored. An example of an off-policy algorithm is Q-learning.

Filter by
Sorted by
Tagged with
2
votes
1answer
31 views

How to prove importance sampling ratio is uncorrelated with action-value (or state-value) estimate?

In Sutton & Barto (2nd edition), the following is mentioned on page 150 (p. 172 of the pdf), section 7.4: the importance sampling ratio has expected value one (Section 5.9) and is uncorrelated ...
2
votes
0answers
25 views

Doubt in Sutton & Barto's off-policy Monte Carlo control algorithm

The algorithm is described as below: My understanding: In the third last step, we act greedily w.r.t $Q$. Since we use importance sampling, this $Q \approx Q_\pi$. However, in the next step, whenever ...
1
vote
1answer
56 views

Which policy has to be followed by a player while construction of its own Q-table?

Consider the scenario, where there are two players. One of the players perform the action randomly, whereas I want second player as a Q-player. I mean, the player selects a best action from the Q-...
0
votes
1answer
65 views

Which policy do I need to use in updating Q function?

Policy function can be of two types: deterministic policy and stochastic policy. Deterministic policy is of the form $\pi : S \rightarrow A$ Stochastic policy is defined using conditional probability ...
2
votes
1answer
58 views

In off-policy MC control algorithm by Sutton & Barto, why do we perform a last update when sample action is inconsistent with target policy?

I have a question about the $W$ term in the off-policy MC control algorithm on Page 111 of Sutton & Barto. I have also included it in the figure below. My question: shouldn't the check $A_{t} = \...
2
votes
2answers
168 views

With Monte Carlo off-policy learning what do we correct by using importance sampling?

I do not understand the link of importance sampling to Monte Carlo off-policy learning. We estimate a value using sampling on whole episodes, and we take these values to construct the target policy. ...
0
votes
0answers
26 views

Which off-policy policy gradient estimator has lower variance?

Let $\pi_\theta$ be a target policy and $\beta_\theta$ be a behavior policy. I have seen the following 2 policy gradient estimators in the literature: $$ \operatorname*{E}_{\tau \sim \beta_\theta} \...
3
votes
1answer
79 views

Why can we take the action $a$ from the next state $s'$ in the max part of the Q-learning update rule, if that action doesn't lead to any reward?

I'm using OpenAI's cartpole environment. First of all, is this environment not Markov? Knowing that, my main question concerns Q-learning and off-policy methods: For me, there is something weird in ...
2
votes
1answer
93 views

Can I add expert data to the replay buffer used by the DDPG algorithm in order to make it converge faster?

I am working on a restricted reinforcement learning environment, i.e. the environment breaks very often (i.e.: the communication between the simulator and reinforcement learning agent breaks after ...
1
vote
0answers
38 views

Understanding policy gradient derivation with importance sampling

I was watching a lecture on Policy Gradient and had difficulty following it when importance sampling was introduced. It was shown that the gradient of the objective can be written as $$\nabla_\theta U(...
2
votes
1answer
145 views

Offline/Batch Reinforcement Learning: when to stop training and what agent to select

Context: My team and I are working on a RL problem for a specific application. We have data collected from user interactions (states, actions, rewards, etc.). It is too costly for us to emulate agents....
5
votes
1answer
162 views

Why do we need importance sampling?

I was studying the off-policy policy improvement method. Then I encountered importance sampling. I completely understood the mathematics behind the calculation, but I am wondering what is the ...
3
votes
1answer
95 views

Why does off-policy learning outperform on-policy learning?

I am self-studying about Reinforcement Learning using different online resources. I now have a basic understanding of how RL works. I saw this in a book: Q-learning is an off-policy learner. An off-...
2
votes
1answer
88 views

When learning off-policy with multi-step returns, why do we use the current behaviour policy in importance sampling?

When learning off-policy with multi-step returns, we want to update the value of $Q(s_1, a_1)$ using rewards from the trajectory $\tau = (s_1, a_1, r_1, s_2, a_2, r_2, ..., s_n, a_n, r_n, s_n+1)$. We ...
2
votes
0answers
45 views

Is Deep SARSA learning a feasible approach?

I noticed that SARSA has been rarely used in the deep RL setting. Usually, the training for DQN is done off-policy. I think one of the major reasons for this is due to greater sample efficiency in ...
3
votes
0answers
169 views

In Soft Actor Critic, why is the action sampled from current policy instead of replay buffer on value function update?

While reading the original paper of Soft Actor Critic, I came across on page number 5, under equation (5) and (6) $$ J_{V}(\psi)=\mathbb{E}_{\mathbf{s}_{t} \sim \mathcal{D}}\left[\frac{1}{2}\left(V_{\...
0
votes
0answers
44 views

Off-policy full-random training in easy-to-explore environment

Let say we are in an environment where a random agent can easily explore all the states of an environment (for example: tic-tac-toe). In those environments, using off-policy algorithm, is it a good ...
0
votes
1answer
27 views

Learning only using off-policy samples

When training policies, is there a reason we need on-policy samples? For expensive simulations, it makes sense to try and reuse samples. Say we're interested in hyperparameter tuning. Can we collect a ...
1
vote
0answers
36 views

How can I derive n-step off-policy temporal difference formula?

I was reading the book "Reinforcement Learning: An Introduction" by Sutton and Barto. In section 7.3, they write the formula for n-step off-policy TD as $$V(S_t) = V(S_{t-1}) + \alpha \rho_{...
3
votes
1answer
121 views

What is meant by "generate the data" in describing the difference between on-policy and off-policy?

From the book: Sutton, Richard S.,Barto, Andrew G.. Reinforcement Learning (Adaptive Computation and Machine Learning series) (p. 100). The MIT Press. Kindle Edition. " following is stated: "...
1
vote
0answers
30 views

Should the importance sampling ratio be updated at the end of the for loop in the off-policy Monte Carlo control algorithm?

I'm studying RL with Sutton and Barto's book. I'd like to ask about the order of execution of a statement in the algorithm below. Here, $W$ (importance sampling ratio) is updated at the end of the <...
1
vote
0answers
42 views

Does importance sampling for off-policy estimation also apply to the case of negative rewards?

Importance sampling is a common method for calculating off-policy estimates in RL. I have been reading through some of the original documentation (D.G. Horvitz and D.J. Thompson, Powell, M.J. and ...
3
votes
0answers
33 views

Why is it the case that off-policy evaluation using importance sampling suffers from high variance?

The average return for trajectories, $V^{\pi_e}$(s) is often computed via the importance sampling estimate $$V^{\pi_e}(s) = \frac{1}{n}\sum_{i=1}^n\prod_{t=0}^{H}\frac{\pi_e(a_t | s_t)}{\pi_b(a_t|s_t)}...
1
vote
0answers
51 views

Is this figure a correct representation of off-policy actor-critic methods?

Does this figure correctly represent the overall general idea about actor-critic methods for on-policy (left) and off-policy (right) case? I am a bit confused about the off-policy case (right figure). ...
1
vote
0answers
77 views

Does the off-policy evaluation work for non-stationary policies?

As the title says, in reinforcement learning, does the off-policy evaluation work for non-stationary policies? For example, IS (importance sampling)-based estimators, such as weighted IS or doubly ...
0
votes
1answer
86 views

How do I know that the DQN has learnt an appropriate Q function?

Is there any sanity check to know whether the Q functions learnt are appropriate in deep Q networks? I know that the Q values for end states should approximate the terminal reward. However, is it ...
2
votes
2answers
383 views

How is per-decision importance sampling derived in Sutton & Barto's book?

In per-decison importance sampling given in Sutton & Barto's book: Eq 5.12 $\rho_{t:T-1}R_{t+k} = \frac{\pi(A_{t}|S_{t})}{b(A_{t}|S_{t})}\frac{\pi(A_{t+1}|S_{t+1})}{b(A_{t+1}|S_{t+1})}\frac{\pi(...
2
votes
0answers
49 views

Can weighted importance sampling be applied to off-policy evaluation for continuous state space MDPs?

Can weighted importance sampling (WIS) and importance sampling (IS) be applied to off-policy evaluation for continuous state spaces MDPs? Given that I have trajectories of $(s_t,a_t)$ pairs and the ...
1
vote
0answers
41 views

How can I perform policy update in python? [closed]

I'm using Python and tensorflow to implement a Deep Q-learning with experience replay in a continous action and state spaces and I have used two neural networks to approximate both the policy function ...
2
votes
3answers
159 views

What is the intuition behind importance sampling for off-policy value evaluation?

The technique for off-policy value evaluation comes from importance sampling, which states that $$E_{x \sim q}[f(x)] \approx \frac{1}{n}\sum_{i=1}^n f(x_i)\frac{q(x_i)}{p(x_i)},$$ where $x_i$ is ...
3
votes
1answer
111 views

What is the difference between on-policy and off-policy for continuous environments?

I'm trying to understand RL applied to time series (so with infinite horizon) which have a continous state space and a discrete action space. First, some preliminary questions: in this case, what is ...
1
vote
1answer
66 views

Is it possible to prove that the target policy is better than the behavioural policy based on learned Q values?

I have retrospective data for a sort of "behaviour policy" which I will use to train a deep q network to learn a target greedy policy. After learning the Q values for this target policy, can we make ...
1
vote
1answer
122 views

Can we combine Off-Policy with On-Policy Algorithms?

On-Policy Algorithms like PPO directly maximize the performance objective or an approximation of it. They tend to be quite stable and reliable but are often sample inefficient. Off-Policy Algorithms ...
2
votes
2answers
428 views

Why is DDPG an off-policy RL algorithm?

In DDPG, if there are no $\epsilon$-greedy and no action noise, is DDPG an on-policy algorithm?
2
votes
3answers
65 views

How to estimate a behavior policy for off-policy learning based on data?

I have a dataset which includes states, actions, and reward. The dataset includes information on the transition, i.e., $p(r,s' \mid s,a)$. Is there a way to estimate a behavior policy from this ...
1
vote
1answer
280 views

What are the differences between 1-step SARSA and SARSA?

SARSA is on-policy, while n-step SARSA is off-policy. But when n = 1, is it like an off-policy version of SARSA? Any similarity and difference between 1-step SARSA and SARSA?
1
vote
1answer
54 views

Understanding the W term in off policy monte carlo learning

In Sutton and Barto's RL textbook they included the following pseudocode for off policy Monte Carlo learning. I am a little confused, however, because to me it looks like the W term will become ...
5
votes
1answer
977 views

Is Expected SARSA an off-policy or on-policy algorithm?

I understand that SARSA is an On-policy algorithm, and Q-learning an off-policy one. Sutton and Barto's textbook describes Expected Sarsa thusly: In these cliff walking results Expected Sarsa was ...
2
votes
1answer
59 views

Could we update the policy network with previous trajectories using supervised learning?

I believe to understand the reason why on-policy methods cannot reuse trajectories collected from earlier policies: the trajectory distribution change with the policy and the policy gradient is ...
2
votes
1answer
36 views

Equivalence between expected parameter increments in "Off-Policy Temporal-Difference Learning with Function Approximation"

I am having a hard time understanding the proof of theorem 1 presented in the "Off-Policy Temporal-Difference Learning with Function Approximation" paper. Let $\Delta \theta$ and $\Delta \...
2
votes
1answer
743 views

Are model-free and off-policy algorithms the same?

In respect of RL, is model-free and off-policy the same thing, just different terminology? If not, what are the differences? I've read that the policy can be thought of as 'the brain', or decision ...
2
votes
0answers
49 views

Can the importance sampling estimator have a non-stationary behaviour policy even if the target policy is stationary?

The inverse propensity score (IPS) estimator, which is used for off-policy evaluation in a contextual bandit problem, is well explained in the paper Doubly Robust Policy Evaluation and Optimization. ...
3
votes
1answer
85 views

What is a non-starving policy in reinforcement learning?

In the paper, Eligibility Traces for off-Policy Policy Evaluation (2010), by Doina Precup et al., mentioned the term "non-starving" many times. The specific use of the term was like "non-starving ...
1
vote
1answer
36 views

Do I need to store the policy for RL?

I am creating a zero-sum game with RL and wondered if I need to store the policy, or if there are other RL methods that produce similar results (consistently beating the human player) without the need ...
4
votes
1answer
221 views

Understanding the n-step off-policy SARSA update

In Sutton & Barto's book (2nd ed) page 149, there is the equation 7.11 I am having a hard time understanding this equation. I would have thought that we should be moving $Q$ towards $G$, where $...
2
votes
1answer
75 views

On-policy distribution for Emphatic TD

The book by Sutton and Barto discussed in section 11.8 that the convergence of off-policy TD function approximation can be improved by correcting for distribution of states encountered. The section ...
8
votes
1answer
3k views

What is the relation between online (or offline) learning and on-policy (or off-policy) algorithms?

In the context of RL, there is the notion of on-policy and off-policy algorithms. I understand the difference between on-policy and off-policy algorithms. Moreover, in RL, there's also the notion of ...
6
votes
1answer
125 views

How do I compute the variance of the return of an evaluation policy using two behaviour policies?

Suppose there is an evaluation policy called $\pi_{e}$ and there are two behavior policies $\pi_{b1}$ and $\pi_{b2}$. I know that it is possible to estimate the return of policy $\pi_{e}$ through ...
5
votes
2answers
292 views

How can the importance sampling ratio be different than zero when the target policy is deterministic?

In the book Reinforcement Learning: An Introduction (2nd edition) Sutton and Barto define at page 104 (p. 126 of the pdf), equation (5.3), the importance sampling ratio, $\rho _{t:T-1}$, as follows: $$...
2
votes
1answer
165 views

Why is the actor-critic algorithm limited to using on-policy data?

Why is the actor-critic algorithm limited to using on-policy data? Or can we use the actor-critic algorithm with off-policy data?