# Questions tagged [importance-sampling]

For questions related to the concept of importance sampling (which comes up, for example, in reinforcement learning).

32 questions
Filter by
Sorted by
Tagged with
43 views

### Derive Importance Sampling as Expected Value Notation

I'm new to RL. Recently, I took a course on Coursera. In the Off-policy MC method, I learned the concept of Importance Sampling as follows: where the importance sampling ratio is the ratio of the ...
31 views

### How to prove importance sampling ratio is uncorrelated with action-value (or state-value) estimate?

In Sutton & Barto (2nd edition), the following is mentioned on page 150 (p. 172 of the pdf), section 7.4: the importance sampling ratio has expected value one (Section 5.9) and is uncorrelated ...
52 views

73 views

162 views

### Why do we need importance sampling?

I was studying the off-policy policy improvement method. Then I encountered importance sampling. I completely understood the mathematics behind the calculation, but I am wondering what is the ...
29 views

### How to compute the Retrace target for multi-step off-policy Reinforcement Learning?

I am implementing the A3C algorithm and I want to add off-policy training using Retrace but I am having some trouble understanding how to compute the retrace target. Retrace is used in combination ...
88 views

### When learning off-policy with multi-step returns, why do we use the current behaviour policy in importance sampling?

When learning off-policy with multi-step returns, we want to update the value of $Q(s_1, a_1)$ using rewards from the trajectory $\tau = (s_1, a_1, r_1, s_2, a_2, r_2, ..., s_n, a_n, r_n, s_n+1)$. We ...
36 views

77 views

### Does the off-policy evaluation work for non-stationary policies?

As the title says, in reinforcement learning, does the off-policy evaluation work for non-stationary policies? For example, IS (importance sampling)-based estimators, such as weighted IS or doubly ...
266 views

### Why we don't use importance sampling in tabular Q-Learning?

Why don't we use an importance sampling ratio in Q-Learning, even though Q-Learning is an off-policy method? Importance sampling is used to calculate expectation of a random variable by using data ...