Questions tagged [off-policy-methods]
For questions related to off-policy reinforcement learning algorithms, which estimate a policy (the target policy) while using another policy (the behavior policy), during the learning process, which ensures that all states are sufficiently explored. An example of an off-policy algorithm is Q-learning.
70
questions
1
vote
1
answer
87
views
How would one normalize observations in off-policy online reinforcement learning?
In off-policy algorithms such as DQN, you need to feed your input to a network twice. 1. When inputting into a network for predicting the Q values. 2. When feeding the input from the buffer to the ...
- 588
0
votes
1
answer
30
views
Fit Q Evaluation in offline reinforcement learning
I am working on a PyTorch implementation of Implicit Q-Learning (IQL) (paper), given a dataset $\mathcal D = \left\{ (\mathbf s_i, \mathbf a_i, \mathbf s_i', r_i ) \right\}$ of transitions. I think I ...
- 101
1
vote
2
answers
41
views
Why does HER not work with on-policy RL algorithms?
I'm wondering because I don't appreciate what is wrong with just applying HER to an otherwise on-policy algorithm? Like if we do that will the training stability just fall apart? And if so why? My ...
- 360
1
vote
1
answer
46
views
Are on-policy algorithms always better than off-policy ones?
I am studying RL and I have a question: Are on-policy algorithms always better than off-policy ones?
1
vote
0
answers
41
views
Prioritized experience replay correction with off-policy estimators
Prioritized exeperience replay (PER) biases the sampling and introduces importance sampling (IS) correction to the Q-function update.
Weights are $w = \frac{1}{N P}^\beta$, where $N$ is the batch size ...
- 153
1
vote
1
answer
145
views
How does off-policy Monte Carlo weighted importance sampling bias converge to zero (Sutton & Barto Section 5.5)
On Section 5.5 (page 105) of Sutton & Barto's "Reinforcement Learning: An Introduction", they discuss the off-policy Monte Carlo method for learning the value function of a target policy ...
- 208
0
votes
0
answers
45
views
Are there papers that do an empirical investigation on DRL hyperparameters?
I am looking for papers that perform a study on DRL hyper-parameters. This paper does a fantastic job of describing the hyperparameters for on-policy algorithms. It would be great to get similar ...
- 588
0
votes
0
answers
31
views
How to evaluate the performance of off-line & model-free reinforcement leaning?
I'm currently studying on off-line reinforcement learning (RL) and trying to utilize it for medical data. Because it seemed hard to develop well-performing environment model, I decided to adopt model-...
- 1
1
vote
0
answers
41
views
Policy Gradient Methods when using a fixed initial sequence of actions
I am implementing a Policy Gradient agent based on IMPALA. Specifically, I'm working on DeepNash, but that is not considerably different from vanilla IMPALA for the purposes of this question.
In my ...
1
vote
1
answer
172
views
In off-policy MC learning, why is the probability of sampling a trajectory the same as having a return?
In Sutton and Barto's RL book, in the section for off-policy learning, we would like to find the expected value of the random variable $G_t$, given $S_t = s$ under our target policy: $$\mathbb{E}_{\pi}...
- 13
1
vote
0
answers
298
views
Why Soft Actor-Critic (SAC) uses a double Q trick?
Twin Delayed DDPG (TD3) uses a double Q trick since the policy is deterministic like in DDPG, which is to mitigate the maximum overestimation bias in DDPG. However, in SAC, the policy is stochastic, ...
- 41
3
votes
0
answers
60
views
How does off-policy monte carlo explore and converge? [duplicate]
Premises to question:
Behavior Policy: e-greedy (stochastic)
Target Policy: greedy (deterministic)
Importance Sampling Included
In off-policy Monte-Carlo control, the behavior policy chooses actions ...
- 31
0
votes
0
answers
39
views
Why would it be easy to evaluate a probability, when it is hard to sample from for importance sampling?
Suppose we want to perform importance sampling where we have trajectories from some behavioral policy $b$, but we want to perform off-policy evaluation. From these prior questions, I understand that ...
- 101
0
votes
1
answer
69
views
Has there been a study done in tuning hyper-parameters for off-policy reinforcement learning?
I am interested in learning about hyper-parameter tuning for off-policy reinforcement learning (specifically DQN). Could someone point me to papers published or empirical observations in this area?
- 588
2
votes
0
answers
116
views
Why do off-policy algorithms suffer from worse computational or time efficiency compared to on-policy algorithms?
When I run Soft-Actor-Critic (off-policy) in my Environment, the calculation of gradient updates takes almost twice the time compared to using PPO (on-policy).
I also saw that ACER has a higher time ...
- 115
2
votes
1
answer
227
views
Can off-policy algorithms benefit from the parallelization?
On-policy algorithms, such as A2C, A3C and PPO, leverage massive parallelization to achieve state of the art results. However, I’ve never come across parallelization efforts when it comes to the off-...
- 331
1
vote
0
answers
67
views
Why does importance sampling ratio start and end one step later in off-policy SARSA given in Sutton-Barto's RL book?
In Sutton & Barto's RL book (page 149) they say:
Sarsa update can be completely replaced by a simple off-policy form
$Q_{t+n}(S_t,A_t)=Q_{t+n−1}(S_t,A_t) + \rho_{t+1:t+n} [G_{t:t+n} − Q_{t+n−1}(...
- 301
1
vote
0
answers
62
views
Proper way to count environment steps / frames in distributed RL architecture for algorithms like CLEAR or LASER => modified impala with replay
In classical - on-policy - vtrace/Impala algorithm env_steps are incremented every training iteration
like this : ...
- 11
2
votes
1
answer
363
views
How does this TD(0) off-policy value update formula work?
The update formula for the TD(0) off-policy learning algorithm is (taken from these slides by D. Silver for lecture 5 of his course)
$$ \underbrace{V(S_t)}_{\text{New value}} \leftarrow \underbrace{V(...
- 133
2
votes
1
answer
54
views
How to prove importance sampling ratio is uncorrelated with action-value (or state-value) estimate?
In Sutton & Barto (2nd edition), the following is mentioned on page 150 (p. 172 of the pdf), section 7.4:
the importance sampling ratio has expected value one (Section 5.9) and is uncorrelated ...
- 359
2
votes
0
answers
88
views
Doubt in Sutton & Barto's off-policy Monte Carlo control algorithm
The algorithm is described as below:
My understanding: In the third last step, we act greedily w.r.t $Q$. Since we use importance sampling, this $Q \approx Q_\pi$. However, in the next step, whenever ...
- 359
1
vote
1
answer
70
views
Which policy has to be followed by a player while construction of its own Q-table?
Consider the scenario, where there are two players. One of the players perform the action randomly, whereas I want second player as a Q-player. I mean, the player selects a best action from the Q-...
- 187
0
votes
1
answer
300
views
Which policy do I need to use in updating Q function?
Policy function can be of two types: deterministic policy and stochastic policy.
Deterministic policy is of the form $\pi : S \rightarrow A$
Stochastic policy is defined using conditional probability ...
- 187
2
votes
1
answer
201
views
In off-policy MC control algorithm by Sutton & Barto, why do we perform a last update when sample action is inconsistent with target policy?
I have a question about the $W$ term in the off-policy MC control algorithm on Page 111 of Sutton & Barto. I have also included it in the figure below.
My question: shouldn't the check $A_{t} = \...
- 123
2
votes
2
answers
281
views
With Monte Carlo off-policy learning what do we correct by using importance sampling?
I do not understand the link of importance sampling to Monte Carlo off-policy learning.
We estimate a value using sampling on whole episodes, and we take these values to construct the target policy.
...
- 221
3
votes
1
answer
139
views
Why can we take the action $a$ from the next state $s'$ in the max part of the Q-learning update rule, if that action doesn't lead to any reward?
I'm using OpenAI's cartpole environment. First of all, is this environment not Markov?
Knowing that, my main question concerns Q-learning and off-policy methods:
For me, there is something weird in ...
- 155
2
votes
1
answer
323
views
Can I add expert data to the replay buffer used by the DDPG algorithm in order to make it converge faster?
I am working on a restricted reinforcement learning environment, i.e. the environment breaks very often (i.e.: the communication between the simulator and reinforcement learning agent breaks after ...
2
votes
1
answer
810
views
Offline/Batch Reinforcement Learning: when to stop training and what agent to select
Context:
My team and I are working on a RL problem for a specific application. We have data collected from user interactions (states, actions, rewards, etc.).
It is too costly for us to emulate agents....
- 21
5
votes
1
answer
1k
views
Why do we need importance sampling?
I was studying the off-policy policy improvement method. Then I encountered importance sampling. I completely understood the mathematics behind the calculation, but I am wondering what is the ...
5
votes
1
answer
192
views
Why does off-policy learning outperform on-policy learning?
I am self-studying about Reinforcement Learning using different online resources. I now have a basic understanding of how RL works.
I saw this in a book:
Q-learning is an off-policy learner. An off-...
- 253
3
votes
1
answer
163
views
When learning off-policy with multi-step returns, why do we use the current behaviour policy in importance sampling?
When learning off-policy with multi-step returns, we want to update the value of $Q(s_1, a_1)$ using rewards from the trajectory $\tau = (s_1, a_1, r_1, s_2, a_2, r_2, ..., s_n, a_n, r_n, s_n+1)$. We ...
- 233
2
votes
0
answers
105
views
Is Deep SARSA learning a feasible approach?
I noticed that SARSA has been rarely used in the deep RL setting. Usually, the training for DQN is done off-policy. I think one of the major reasons for this is due to greater sample efficiency in ...
- 1,251
3
votes
0
answers
517
views
In Soft Actor Critic, why is the action sampled from current policy instead of replay buffer on value function update?
While reading the original paper of Soft Actor Critic, I came across on page number 5, under equation (5) and (6)
$$
J_{V}(\psi)=\mathbb{E}_{\mathbf{s}_{t} \sim \mathcal{D}}\left[\frac{1}{2}\left(V_{\...
- 31
0
votes
0
answers
53
views
Off-policy full-random training in easy-to-explore environment
Let say we are in an environment where a random agent can easily explore all the states of an environment (for example: tic-tac-toe).
In those environments, using off-policy algorithm, is it a good ...
- 266
0
votes
1
answer
35
views
Learning only using off-policy samples
When training policies, is there a reason we need on-policy samples? For expensive simulations, it makes sense to try and reuse samples. Say we're interested in hyperparameter tuning. Can we collect a ...
- 101
1
vote
0
answers
89
views
How can I derive n-step off-policy temporal difference formula?
I was reading the book "Reinforcement Learning: An Introduction" by Sutton and Barto. In section 7.3, they write the formula for n-step off-policy TD as
$$V(S_t) = V(S_{t-1}) + \alpha \rho_{...
- 673
3
votes
1
answer
152
views
What is meant by "generate the data" in describing the difference between on-policy and off-policy?
From the book:
Sutton, Richard S.,Barto, Andrew G.. Reinforcement Learning (Adaptive Computation and Machine Learning series) (p. 100). The MIT Press. Kindle Edition. "
following is stated:
"...
- 325
1
vote
0
answers
72
views
Should the importance sampling ratio be updated at the end of the for loop in the off-policy Monte Carlo control algorithm?
I'm studying RL with Sutton and Barto's book. I'd like to ask about the order of execution of a statement in the algorithm below.
Here, $W$ (importance sampling ratio) is updated at the end of the <...
- 11
1
vote
0
answers
114
views
Does importance sampling for off-policy estimation also apply to the case of negative rewards?
Importance sampling is a common method for calculating off-policy estimates in RL. I have been reading through some of the original documentation (D.G. Horvitz and D.J. Thompson, Powell, M.J. and ...
- 33
3
votes
0
answers
59
views
Why is it the case that off-policy evaluation using importance sampling suffers from high variance?
The average return for trajectories, $V^{\pi_e}$(s) is often computed via the importance sampling estimate $$V^{\pi_e}(s) = \frac{1}{n}\sum_{i=1}^n\prod_{t=0}^{H}\frac{\pi_e(a_t | s_t)}{\pi_b(a_t|s_t)}...
- 1,251
1
vote
0
answers
67
views
Is this figure a correct representation of off-policy actor-critic methods?
Does this figure correctly represent the overall general idea about actor-critic methods for on-policy (left) and off-policy (right) case?
I am a bit confused about the off-policy case (right figure). ...
- 11
2
votes
0
answers
110
views
Does the off-policy evaluation work for non-stationary policies?
As the title says, in reinforcement learning, does the off-policy evaluation work for non-stationary policies?
For example, IS (importance sampling)-based estimators, such as weighted IS or doubly ...
- 227
0
votes
1
answer
195
views
How do I know that the DQN has learnt an appropriate Q function?
Is there any sanity check to know whether the Q functions learnt are appropriate in deep Q networks? I know that the Q values for end states should approximate the terminal reward. However, is it ...
- 1,251
2
votes
2
answers
738
views
How is per-decision importance sampling derived in Sutton & Barto's book?
In per-decison importance sampling given in Sutton & Barto's book:
Eq 5.12 $\rho_{t:T-1}R_{t+k} = \frac{\pi(A_{t}|S_{t})}{b(A_{t}|S_{t})}\frac{\pi(A_{t+1}|S_{t+1})}{b(A_{t+1}|S_{t+1})}\frac{\pi(...
- 147
2
votes
0
answers
81
views
Can weighted importance sampling be applied to off-policy evaluation for continuous state space MDPs?
Can weighted importance sampling (WIS) and importance sampling (IS) be applied to off-policy evaluation for continuous state spaces MDPs?
Given that I have trajectories of $(s_t,a_t)$ pairs and the ...
- 1,251
3
votes
3
answers
941
views
What is the intuition behind importance sampling for off-policy value evaluation?
The technique for off-policy value evaluation comes from importance sampling, which states that
$$E_{x \sim q}[f(x)] \approx \frac{1}{n}\sum_{i=1}^n f(x_i)\frac{q(x_i)}{p(x_i)},$$ where $x_i$ is ...
- 1,251
4
votes
1
answer
271
views
What is the difference between on-policy and off-policy for continuous environments?
I'm trying to understand RL applied to time series (so with infinite horizon) which have a continous state space and a discrete action space.
First, some preliminary questions: in this case, what is ...
- 331
1
vote
1
answer
73
views
Is it possible to prove that the target policy is better than the behavioural policy based on learned Q values?
I have retrospective data for a sort of "behaviour policy" which I will use to train a deep q network to learn a target greedy policy. After learning the Q values for this target policy, can we make ...
- 1,251
1
vote
1
answer
457
views
Can we combine Off-Policy with On-Policy Algorithms?
On-Policy Algorithms like PPO directly maximize the performance objective or an approximation of it. They tend to be quite stable and reliable but are often sample inefficient. Off-Policy Algorithms ...
- 451
2
votes
2
answers
1k
views
Why is DDPG an off-policy RL algorithm?
In DDPG, if there are no $\epsilon$-greedy and no action noise, is DDPG an on-policy algorithm?
- 150