Questions tagged [off-policy-methods]

For questions related to off-policy reinforcement learning algorithms, which estimate a policy (the target policy) while using another policy (the behavior policy), during the learning process, which ensures that all states are sufficiently explored. An example of an off-policy algorithm is Q-learning.

Filter by
Sorted by
Tagged with
2 votes
1 answer
31 views

Why is there the potential problem of "learning only from the tails of episodes" in off-policy MC control?

Sutton-Barto page 111, first paragraph (Off-policy Monte Carlo Control): A potential problem is that this method learns only from the tails of episodes, when all of the remaining actions in the ...
DSPinfinity's user avatar
0 votes
1 answer
24 views

A question about the reward calculation in the Hindsight Experience Replay algorithm

I' m try to implement the HER algorithm from scratch in order to use it in the PandaReach-v3 environment. I already developed the same algorithm for the bitflip environment and it works as expected. ...
Dave's user avatar
  • 155
1 vote
1 answer
147 views

How would one normalize observations in off-policy online reinforcement learning?

In off-policy algorithms such as DQN, you need to feed your input to a network twice. 1. When inputting into a network for predicting the Q values. 2. When feeding the input from the buffer to the ...
desert_ranger's user avatar
2 votes
1 answer
231 views

Fit Q Evaluation in offline reinforcement learning

I am working on a PyTorch implementation of Implicit Q-Learning (IQL) (paper), given a dataset $\mathcal D = \left\{ (\mathbf s_i, \mathbf a_i, \mathbf s_i', r_i ) \right\}$ of transitions. I think I ...
Novice's user avatar
  • 121
1 vote
2 answers
55 views

Why does HER not work with on-policy RL algorithms?

I'm wondering because I don't appreciate what is wrong with just applying HER to an otherwise on-policy algorithm? Like if we do that will the training stability just fall apart? And if so why? My ...
profPlum's user avatar
  • 424
1 vote
1 answer
65 views

Are on-policy algorithms always better than off-policy ones?

I am studying RL and I have a question: Are on-policy algorithms always better than off-policy ones?
Samvel Safaryan's user avatar
1 vote
0 answers
41 views

Prioritized experience replay correction with off-policy estimators

Prioritized exeperience replay (PER) biases the sampling and introduces importance sampling (IS) correction to the Q-function update. Weights are $w = \frac{1}{N P}^\beta$, where $N$ is the batch size ...
Simon's user avatar
  • 153
2 votes
1 answer
232 views

How does off-policy Monte Carlo weighted importance sampling bias converge to zero (Sutton & Barto Section 5.5)

On Section 5.5 (page 105) of Sutton & Barto's "Reinforcement Learning: An Introduction", they discuss the off-policy Monte Carlo method for learning the value function of a target policy ...
user118967's user avatar
0 votes
0 answers
37 views

How to evaluate the performance of off-line & model-free reinforcement leaning?

I'm currently studying on off-line reinforcement learning (RL) and trying to utilize it for medical data. Because it seemed hard to develop well-performing environment model, I decided to adopt model-...
Maverick's user avatar
1 vote
0 answers
46 views

Policy Gradient Methods when using a fixed initial sequence of actions

I am implementing a Policy Gradient agent based on IMPALA. Specifically, I'm working on DeepNash, but that is not considerably different from vanilla IMPALA for the purposes of this question. In my ...
Ryan Keathley's user avatar
1 vote
1 answer
224 views

In off-policy MC learning, why is the probability of sampling a trajectory the same as having a return?

In Sutton and Barto's RL book, in the section for off-policy learning, we would like to find the expected value of the random variable $G_t$, given $S_t = s$ under our target policy: $$\mathbb{E}_{\pi}...
ArminAshrafi's user avatar
1 vote
0 answers
459 views

Why Soft Actor-Critic (SAC) uses a double Q trick?

Twin Delayed DDPG (TD3) uses a double Q trick since the policy is deterministic like in DDPG, which is to mitigate the maximum overestimation bias in DDPG. However, in SAC, the policy is stochastic, ...
Magi Feeney's user avatar
3 votes
0 answers
60 views

How does off-policy monte carlo explore and converge? [duplicate]

Premises to question: Behavior Policy: e-greedy (stochastic) Target Policy: greedy (deterministic) Importance Sampling Included In off-policy Monte-Carlo control, the behavior policy chooses actions ...
Jonah Kim's user avatar
0 votes
0 answers
52 views

Why would it be easy to evaluate a probability, when it is hard to sample from for importance sampling?

Suppose we want to perform importance sampling where we have trajectories from some behavioral policy $b$, but we want to perform off-policy evaluation. From these prior questions, I understand that ...
derekchen14's user avatar
0 votes
1 answer
71 views

Has there been a study done in tuning hyper-parameters for off-policy reinforcement learning?

I am interested in learning about hyper-parameter tuning for off-policy reinforcement learning (specifically DQN). Could someone point me to papers published or empirical observations in this area?
desert_ranger's user avatar
2 votes
0 answers
137 views

Why do off-policy algorithms suffer from worse computational or time efficiency compared to on-policy algorithms?

When I run Soft-Actor-Critic (off-policy) in my Environment, the calculation of gradient updates takes almost twice the time compared to using PPO (on-policy). I also saw that ACER has a higher time ...
kitaird's user avatar
  • 115
2 votes
1 answer
331 views

Can off-policy algorithms benefit from the parallelization?

On-policy algorithms, such as A2C, A3C and PPO, leverage massive parallelization to achieve state of the art results. However, I’ve never come across parallelization efforts when it comes to the off-...
Mika's user avatar
  • 341
1 vote
0 answers
86 views

Why does importance sampling ratio start and end one step later in off-policy SARSA given in Sutton-Barto's RL book?

In Sutton & Barto's RL book (page 149) they say: Sarsa update can be completely replaced by a simple off-policy form $Q_{t+n}(S_t,A_t)=Q_{t+n−1}(S_t,A_t) + \rho_{t+1:t+n} [G_{t:t+n} − Q_{t+n−1}(...
DSPinfinity's user avatar
1 vote
0 answers
66 views

Proper way to count environment steps / frames in distributed RL architecture for algorithms like CLEAR or LASER => modified impala with replay

In classical - on-policy - vtrace/Impala algorithm env_steps are incremented every training iteration like this : ...
parradox's user avatar
2 votes
1 answer
479 views

How does this TD(0) off-policy value update formula work?

The update formula for the TD(0) off-policy learning algorithm is (taken from these slides by D. Silver for lecture 5 of his course) $$ \underbrace{V(S_t)}_{\text{New value}} \leftarrow \underbrace{V(...
KoKlA's user avatar
  • 133
2 votes
1 answer
59 views

How to prove importance sampling ratio is uncorrelated with action-value (or state-value) estimate?

In Sutton & Barto (2nd edition), the following is mentioned on page 150 (p. 172 of the pdf), section 7.4: the importance sampling ratio has expected value one (Section 5.9) and is uncorrelated ...
user529295's user avatar
2 votes
0 answers
104 views

Doubt in Sutton & Barto's off-policy Monte Carlo control algorithm

The algorithm is described as below: My understanding: In the third last step, we act greedily w.r.t $Q$. Since we use importance sampling, this $Q \approx Q_\pi$. However, in the next step, whenever ...
user529295's user avatar
1 vote
1 answer
71 views

Which policy has to be followed by a player while construction of its own Q-table?

Consider the scenario, where there are two players. One of the players perform the action randomly, whereas I want second player as a Q-player. I mean, the player selects a best action from the Q-...
satya's user avatar
  • 187
0 votes
1 answer
367 views

Which policy do I need to use in updating Q function?

Policy function can be of two types: deterministic policy and stochastic policy. Deterministic policy is of the form $\pi : S \rightarrow A$ Stochastic policy is defined using conditional probability ...
satya's user avatar
  • 187
2 votes
1 answer
247 views

In off-policy MC control algorithm by Sutton & Barto, why do we perform a last update when sample action is inconsistent with target policy?

I have a question about the $W$ term in the off-policy MC control algorithm on Page 111 of Sutton & Barto. I have also included it in the figure below. My question: shouldn't the check $A_{t} = \...
Curious2learn's user avatar
2 votes
2 answers
310 views

With Monte Carlo off-policy learning what do we correct by using importance sampling?

I do not understand the link of importance sampling to Monte Carlo off-policy learning. We estimate a value using sampling on whole episodes, and we take these values to construct the target policy. ...
Hermes Morales's user avatar
3 votes
1 answer
143 views

Why can we take the action $a$ from the next state $s'$ in the max part of the Q-learning update rule, if that action doesn't lead to any reward?

I'm using OpenAI's cartpole environment. First of all, is this environment not Markov? Knowing that, my main question concerns Q-learning and off-policy methods: For me, there is something weird in ...
JeanMi's user avatar
  • 165
2 votes
1 answer
363 views

Can I add expert data to the replay buffer used by the DDPG algorithm in order to make it converge faster?

I am working on a restricted reinforcement learning environment, i.e. the environment breaks very often (i.e.: the communication between the simulator and reinforcement learning agent breaks after ...
Dheerendra Singh Tomar's user avatar
2 votes
1 answer
1k views

Offline/Batch Reinforcement Learning: when to stop training and what agent to select

Context: My team and I are working on a RL problem for a specific application. We have data collected from user interactions (states, actions, rewards, etc.). It is too costly for us to emulate agents....
MetaHG's user avatar
  • 21
5 votes
1 answer
1k views

Why do we need importance sampling?

I was studying the off-policy policy improvement method. Then I encountered importance sampling. I completely understood the mathematics behind the calculation, but I am wondering what is the ...
Alireza Hosseini's user avatar
5 votes
1 answer
239 views

Why does off-policy learning outperform on-policy learning?

I am self-studying about Reinforcement Learning using different online resources. I now have a basic understanding of how RL works. I saw this in a book: Q-learning is an off-policy learner. An off-...
Exploring's user avatar
  • 343
3 votes
1 answer
210 views

When learning off-policy with multi-step returns, why do we use the current behaviour policy in importance sampling?

When learning off-policy with multi-step returns, we want to update the value of $Q(s_1, a_1)$ using rewards from the trajectory $\tau = (s_1, a_1, r_1, s_2, a_2, r_2, ..., s_n, a_n, r_n, s_n+1)$. We ...
Federico Taschin's user avatar
2 votes
0 answers
117 views

Is Deep SARSA learning a feasible approach?

I noticed that SARSA has been rarely used in the deep RL setting. Usually, the training for DQN is done off-policy. I think one of the major reasons for this is due to greater sample efficiency in ...
calveeen's user avatar
  • 1,261
3 votes
0 answers
583 views

In Soft Actor Critic, why is the action sampled from current policy instead of replay buffer on value function update?

While reading the original paper of Soft Actor Critic, I came across on page number 5, under equation (5) and (6) $$ J_{V}(\psi)=\mathbb{E}_{\mathbf{s}_{t} \sim \mathcal{D}}\left[\frac{1}{2}\left(V_{\...
DannyBoi's user avatar
0 votes
0 answers
53 views

Off-policy full-random training in easy-to-explore environment

Let say we are in an environment where a random agent can easily explore all the states of an environment (for example: tic-tac-toe). In those environments, using off-policy algorithm, is it a good ...
Loheek's user avatar
  • 266
0 votes
1 answer
35 views

Learning only using off-policy samples

When training policies, is there a reason we need on-policy samples? For expensive simulations, it makes sense to try and reuse samples. Say we're interested in hyperparameter tuning. Can we collect a ...
smorad's user avatar
  • 101
1 vote
0 answers
102 views

How can I derive n-step off-policy temporal difference formula?

I was reading the book "Reinforcement Learning: An Introduction" by Sutton and Barto. In section 7.3, they write the formula for n-step off-policy TD as $$V(S_t) = V(S_{t-1}) + \alpha \rho_{...
Swakshar Deb's user avatar
3 votes
1 answer
156 views

What is meant by "generate the data" in describing the difference between on-policy and off-policy?

From the book: Sutton, Richard S.,Barto, Andrew G.. Reinforcement Learning (Adaptive Computation and Machine Learning series) (p. 100). The MIT Press. Kindle Edition. " following is stated: "...
blue-sky's user avatar
  • 335
1 vote
0 answers
83 views

Should the importance sampling ratio be updated at the end of the for loop in the off-policy Monte Carlo control algorithm?

I'm studying RL with Sutton and Barto's book. I'd like to ask about the order of execution of a statement in the algorithm below. Here, $W$ (importance sampling ratio) is updated at the end of the <...
JungYT's user avatar
  • 11
1 vote
0 answers
127 views

Does importance sampling for off-policy estimation also apply to the case of negative rewards?

Importance sampling is a common method for calculating off-policy estimates in RL. I have been reading through some of the original documentation (D.G. Horvitz and D.J. Thompson, Powell, M.J. and ...
curiouscat22's user avatar
3 votes
0 answers
60 views

Why is it the case that off-policy evaluation using importance sampling suffers from high variance?

The average return for trajectories, $V^{\pi_e}$(s) is often computed via the importance sampling estimate $$V^{\pi_e}(s) = \frac{1}{n}\sum_{i=1}^n\prod_{t=0}^{H}\frac{\pi_e(a_t | s_t)}{\pi_b(a_t|s_t)}...
calveeen's user avatar
  • 1,261
1 vote
0 answers
78 views

Is this figure a correct representation of off-policy actor-critic methods?

Does this figure correctly represent the overall general idea about actor-critic methods for on-policy (left) and off-policy (right) case? I am a bit confused about the off-policy case (right figure). ...
Asad Shahid's user avatar
2 votes
0 answers
116 views

Does the off-policy evaluation work for non-stationary policies?

As the title says, in reinforcement learning, does the off-policy evaluation work for non-stationary policies? For example, IS (importance sampling)-based estimators, such as weighted IS or doubly ...
Hunnam 's user avatar
  • 227
0 votes
1 answer
232 views

How do I know that the DQN has learnt an appropriate Q function?

Is there any sanity check to know whether the Q functions learnt are appropriate in deep Q networks? I know that the Q values for end states should approximate the terminal reward. However, is it ...
calveeen's user avatar
  • 1,261
2 votes
2 answers
823 views

How is per-decision importance sampling derived in Sutton & Barto's book?

In per-decison importance sampling given in Sutton & Barto's book: Eq 5.12 $\rho_{t:T-1}R_{t+k} = \frac{\pi(A_{t}|S_{t})}{b(A_{t}|S_{t})}\frac{\pi(A_{t+1}|S_{t+1})}{b(A_{t+1}|S_{t+1})}\frac{\pi(...
ZERO NULLS's user avatar
2 votes
0 answers
91 views

Can weighted importance sampling be applied to off-policy evaluation for continuous state space MDPs?

Can weighted importance sampling (WIS) and importance sampling (IS) be applied to off-policy evaluation for continuous state spaces MDPs? Given that I have trajectories of $(s_t,a_t)$ pairs and the ...
calveeen's user avatar
  • 1,261
3 votes
3 answers
1k views

What is the intuition behind importance sampling for off-policy value evaluation?

The technique for off-policy value evaluation comes from importance sampling, which states that $$E_{x \sim q}[f(x)] \approx \frac{1}{n}\sum_{i=1}^n f(x_i)\frac{q(x_i)}{p(x_i)},$$ where $x_i$ is ...
calveeen's user avatar
  • 1,261
4 votes
1 answer
341 views

What is the difference between on-policy and off-policy for continuous environments?

I'm trying to understand RL applied to time series (so with infinite horizon) which have a continous state space and a discrete action space. First, some preliminary questions: in this case, what is ...
unter_983's user avatar
  • 331
1 vote
1 answer
76 views

Is it possible to prove that the target policy is better than the behavioural policy based on learned Q values?

I have retrospective data for a sort of "behaviour policy" which I will use to train a deep q network to learn a target greedy policy. After learning the Q values for this target policy, can we make ...
calveeen's user avatar
  • 1,261
1 vote
1 answer
533 views

Can we combine Off-Policy with On-Policy Algorithms?

On-Policy Algorithms like PPO directly maximize the performance objective or an approximation of it. They tend to be quite stable and reliable but are often sample inefficient. Off-Policy Algorithms ...
Ray Walker's user avatar