Questions tagged [importance-sampling]
For questions related to the concept of importance sampling (which comes up, for example, in reinforcement learning).
19
questions
1
vote
0answers
23 views
Should the importance sampling ratio be updated at the end of the for loop in the off-policy Monte Carlo control algorithm?
I'm studying RL with Sutton and Barto's book. I'd like to ask about the order of execution of a statement in the algorithm below.
Here, $W$ (importance sampling ratio) is updated at the end of the <...
0
votes
0answers
27 views
accessible subset of data impacts performance
I have a problem with a subset of my data which is as follows:
I can train a model (doesn't matter what, xgboost, BERT, etc., it is a text classification problem), on my data and get a decent ...
0
votes
1answer
29 views
How is trajectory sampling different than normal (importance) sampling in reinforcement learning?
I am using Sutton and Barto's book for Reinforcement Learning.
In Chapter 8, I am having difficulty in understanding the Trajectory Sampling.
I have read the particular section on trajectory sampling (...
2
votes
0answers
26 views
Why is it the case that off - policy evaluation using importance sampling suffers from high variance?
The average return for trajectories, $V^{\pi_e}$(s) is often computed via the importance sampling estimate $$V^{\pi_e}(s) = \frac{1}{n}\sum_{i=1}^n\prod_{t=0}^{H}\frac{\pi_e(a_t | s_t)}{\pi_b(a_t|s_t)}...
0
votes
0answers
40 views
Does the off-policy evaluation work for non-stationary policies?
As the title says, in reinforcement learning, does the off-policy evaluation work for non-stationary policies?
For example, IS (importance sampling)-based estimators, such as weighted IS or doubly ...
2
votes
1answer
75 views
Why we don't use importance sampling in tabular Q-Learning?
Why don't we use an importance sampling ratio in Q-Learning, even though Q-Learning is an off-policy method?
Importance sampling is used to calculate expectation of a random variable by using data ...
1
vote
0answers
64 views
How is per-decision importance sampling derived in Sutton & Barto's book?
In per-decison importance sampling given in Sutton & Barto's book:
Eq 5.12 $\rho_{t:T-1}R_{t+k} = \frac{\pi(A_{t}|S_{t})}{b(A_{t}|S_{t})}\frac{\pi(A_{t+1}|S_{t+1})}{b(A_{t+1}|S_{t+1})}\frac{\pi(...
1
vote
0answers
40 views
Why does the n-step return being zero result in high variance in off policy n-step TD?
In the paragraph given between eq 7.12 and 7.13 in Sutton & Barto's book:
$G_{t:h} = R_{t+1} + G_{t+1:h} , t < h < T$
where $G_{h:h} = V_{h-1}(S_h)$. (Recall that this return is used at ...
2
votes
0answers
45 views
Can weighted importance sampling be applied to off-policy evaluation for continuous state space MDPs?
Can weighted importance sampling (WIS) and importance sampling (IS) be applied to off-policy evaluation for continuous state spaces MDPs?
Given that I have trajectories of $(s_t,a_t)$ pairs and the ...
0
votes
2answers
70 views
What is the intuition behind importance sampling for off-policy value evaluation?
The technique for off-policy value evaluation comes from importance sampling, which states that
$$E_{x \sim q}[f(x)] \approx \frac{1}{n}\sum_{i=1}^n f(x_i)\frac{q(x_i)}{p(x_i)},$$ where $x_i$ is ...
4
votes
2answers
307 views
How can we compute the ratio between the distributions if we don't know one of the distributions?
Here is my understanding of importance sampling. If we have two distributions $p(x)$ and $q(x)$, where we have a way of sampling from $p(x)$ but not from $q(x)$, but we want to compute the expectation ...
3
votes
1answer
46 views
How is the incremental update rule derived from the weighted importance sampling in off-policy Monte Carlo control?
Here's the approximated value using weighted importance sampling
$$
V_{n} \doteq \frac{\sum_{k=1}^{n-1} W_{k} G_{k}}{\sum_{k=1}^{n-1} W_{k}}, \quad n \geq 2
$$
Here's the incremental update rule for ...
1
vote
0answers
25 views
Importance sampling eq. 5 in paper “Residual Energy-based Models for Text Generation”
In the paper "Residual Energy-Based Models for Text Generation" (arXiv), on page 5, they write that equation 5 is an instance of importance sampling.
Equation 5 is:
$$ P(x_t \mid x_{<t}) = P_{LM}(...
2
votes
0answers
35 views
Can the importance sampling estimator have a non-stationary behaviour policy even if the target policy is stationary?
The inverse propensity score (IPS) estimator, which is used for off-policy evaluation in a contextual bandit problem, is well explained in the paper Doubly Robust Policy Evaluation and Optimization.
...
2
votes
0answers
36 views
Can we use imitation learning for on-policy algorithms?
Imitation learning uses experiences of an (expert) agent to train another agent, in my understanding. If I want to use an on-policy algorithm, for example, Proximal Policy Optimization, because of it'...
2
votes
1answer
44 views
Are successive actions independent?
The proof of the consistency of the per-decision importance sampling estimator assumes the independence of
$$\frac{\pi(A_t|S_t)}{b(A_t|S_t)}R_{t+1}\quad\text{ and }\quad \prod_{k=t+1}^{T-1}\frac{\pi(...
1
vote
2answers
199 views
In the context of importance sampling ratio, how is the equation $\mathbb{E}\left[\rho_{t: T-1} G_{t} | S_{t}=s\right]=v_{\pi}(s)$ derived?
When reading the book by Sutton and Barto, I came across the importance sampling ratio.
The first equation, I believe, describes the probability a particular sequence is obtained given the current ...
2
votes
1answer
113 views
Do we need the transition probability function when calculating the importance sampling ratio?
I am reading the book titled "Reinforcement Learning: An Introduction" (by Sutton and Barto). I am at chapter 5, which is about Monte Carlo methods, but now I am quite confused.
There is one thing I ...
18
votes
2answers
6k views
What is sample efficiency, and how can importance sampling be used to achieve it?
For instance, the title of this paper reads: "Sample Efficient Actor-Critic with Experience Replay".
What is sample efficiency, and how can importance sampling be used to achieve it?
