Share Your Experience: Take the 2024 Developer Survey

# Questions tagged [importance-sampling]

For questions related to the concept of importance sampling (which comes up, for example, in reinforcement learning).

43 questions
Filter by
Sorted by
Tagged with
1 vote
34 views

### How does Importance Sampling compare against Policy Regularization in Offline RL?

In offline RL we are usually equipped with a dataset, often also called Replay Buffer, that contains transitions in an episode, that are created by some behavior policy. This behavior policy could ...
• 185
12 views

### Is there a way to mix the Importance weighted VAE with the Beta-VAE?

Is there a way to mix the Importance weighted VAE with the Beta-VAE? It seems that we cannot separate out the KL-term from the IWAE-ELBO, hence we cannot multiply it with Beta. On the other hand, it ...
• 11
22 views

### Derivation of TD(0) with Gradient Correction (TDC)

I'm trying to understand the derivation of TDC from Sutton and Barto, but I'm stuck on the steps between lines 2 and 3. It seems that the importance sampling ratio just "disappears" between ...
66 views

### What happens when the probability of either one of the policies is 0 in Importance Sampling?

I have a general question about the methods that use importance sampling in RL. What happens when the probability of either one of the policies is 0?
• 21
129 views

### Why is $p(x)=\int p(x,z) dz$ intractable for continuous $z$ in VAE?

In VAE we use the importance sampling trick to use $q_\phi(z|x)$ to help maximize $\log p_\theta(x)=\log \int p_\theta(x,z)dz\ge \int q_\phi(z|x)\log \frac{p_\theta(x,z)}{q_\phi(z|x)} dz$. Meanwhile, ...
• 21
241 views

### How does off-policy Monte Carlo weighted importance sampling bias converge to zero (Sutton & Barto Section 5.5)

On Section 5.5 (page 105) of Sutton & Barto's "Reinforcement Learning: An Introduction", they discuss the off-policy Monte Carlo method for learning the value function of a target policy ...
• 248
1 vote
228 views

492 views

• 113
310 views

### With Monte Carlo off-policy learning what do we correct by using importance sampling?

I do not understand the link of importance sampling to Monte Carlo off-policy learning. We estimate a value using sampling on whole episodes, and we take these values to construct the target policy. ...
104 views

• 693
1 vote
85 views

### Should the importance sampling ratio be updated at the end of the for loop in the off-policy Monte Carlo control algorithm?

I'm studying RL with Sutton and Barto's book. I'd like to ask about the order of execution of a statement in the algorithm below. Here, $W$ (importance sampling ratio) is updated at the end of the <...
• 11
1 vote
128 views

### Does importance sampling for off-policy estimation also apply to the case of negative rewards?

Importance sampling is a common method for calculating off-policy estimates in RL. I have been reading through some of the original documentation (D.G. Horvitz and D.J. Thompson, Powell, M.J. and ...
1 vote
476 views

### How is trajectory sampling different than normal (importance) sampling in reinforcement learning?

I am using Sutton and Barto's book for Reinforcement Learning. In Chapter 8, I am having difficulty in understanding the Trajectory Sampling. I have read the particular section on trajectory sampling (...
• 393
61 views

73 views

### Can the importance sampling estimator have a non-stationary behaviour policy even if the target policy is stationary?

The inverse propensity score (IPS) estimator, which is used for off-policy evaluation in a contextual bandit problem, is well explained in the paper Doubly Robust Policy Evaluation and Optimization. ...
• 227
75 views

### Can we use imitation learning for on-policy algorithms?

Imitation learning uses experiences of an (expert) agent to train another agent, in my understanding. If I want to use an on-policy algorithm, for example, Proximal Policy Optimization, because of it'...
72 views

• 311
1 vote
269 views

### In the context of importance sampling ratio, how is the equation $\mathbb{E}\left[\rho_{t: T-1} G_{t} | S_{t}=s\right]=v_{\pi}(s)$ derived?

When reading the book by Sutton and Barto, I came across the importance sampling ratio. The first equation, I believe, describes the probability a particular sequence is obtained given the current ...
• 113
322 views

### Do we need the transition probability function when calculating the importance sampling ratio?

I am reading the book titled "Reinforcement Learning: An Introduction" (by Sutton and Barto). I am at chapter 5, which is about Monte Carlo methods, but now I am quite confused. There is one thing I ...
2k views

### Why is the log probability replaced with the importance sampling in the loss function?

In the Trust-Region Policy Optimisation (TRPO) algorithm (and subsequently in PPO also), I do not understand the motivation behind replacing the log probability term from standard policy gradients L^...
• 106