0
$\begingroup$

The technique for off-policy value evaluation comes from importance sampling, which states that

$$E_{x \sim q}[f(x)] \approx \frac{1}{n}\sum_{i=1}^n f(x_i)\frac{q(x_i)}{p(x_i)},$$ where $x_i$ is sampled from $p$.

In the application of importance sampling to RL, is the expectation of the function $f$ equivalent to the value of the trajectories, which is represented by the trajectories $x$?

The distributions $p$ represent the probability of sampling trajectories from the behavior policy and the distribution $q$ represents the probability of sampling trajectories from the target policy $q$?

How would the trajectories from distribution $q$ be better than that of $p$? I know from the equation how it is better, but it is hard to understand intuitively why this could be so.

$\endgroup$
  • 1
    $\begingroup$ I think you may have your $p$ and $q$ mixed up here in your original equation. If I understand correctly then you want to evaluate policy $q$ using samples from $p$ ? if so then you would want $\mathbb{E}_{x \sim q}[f(x)]$ but using importance sampling to generate trajectories from $p$ then this would be equivalent to $\mathbb{E}_{x \sim p}[f(x) \frac{q(x)}{p(x)}]$. $\endgroup$ – David Ireland May 22 at 10:44
  • $\begingroup$ Hi David I think you are right i got it mixed up ! $\endgroup$ – calveeen May 22 at 11:00
  • $\begingroup$ could you also elaborate on this sentence - 'is it correct to equate the expectation of the function 𝑓 is equivalent to the value of the trajectories, which is represented by the trajectories 𝑥' please? I'm not quite sure what you mean. $\endgroup$ – David Ireland May 22 at 11:04
  • $\begingroup$ Sorry i didn't explain it that well. I wanted to know how the terms in importance sampling are being applied in the off policy evaluation RL context. $\endgroup$ – calveeen May 22 at 11:06
2
$\begingroup$

Recall that our goal is to be able to accurately estimate the true value of each state by computing a sample average over returns starting from that state: $$v_{q}(s) \doteq \mathbb{E}_{q}\left[G_{t} | S_{t}=s\right] \approx \frac{1}{n} \sum_{i=1}^{n} Return_i $$ where $Return_i$ is the return obtained from the $i^{th}$ trajectory.

The problem is that the $\approx $ does not hold, since in off-policy learning, we got those returns by following the behavior policy, $p$, and not the target policy, $q$.

To address that, we have to correct each return in the sample average by multiplying by the importance sampling ratio.

$$v_{q}(s) \doteq \mathbb{E}_{q}\left[G_{t} | S_{t}=s\right] \approx \frac{1}{n} \sum_{i=1}^{n} \rho_i Return_i$$

where the importance sampling ratio is : $\rho=\frac{\mathbb{P}(\text { trajectory under } q)}{\mathbb{P}(\text { trajectory under } p)}$

What this multiplication does is that it increases the importance of returns that were more likely to be seen under the target policy $q$ and it decreases those that were less likely. So, at the end, in expectation, it would be as if the returns were averaged following $q$.

(A side note: To avoid the risks of mixing $p$ and $q$, it might be a good idea to denote/think of the behavior policy as $b$ and the target policy as $\pi$, following the convention in Sutton and Barto's RL book.)

| improve this answer | |
$\endgroup$
1
$\begingroup$

In the application of importance sampling to RL, is the expectation of the function $f$ equivalent to the value of the trajectories, which is represented by the trajectories $x$?

I believe what you are asking here is if when using importance sampling in the off-policy RL setting that we set $f(x)$ from the general importance sampling formula to be our returns - the answer to this is yes. As always we are interested in calculating our expected returns.

How would the trajectories from the distribution $q$ be better than that of $p$? I know from the equation how it is better but it is hard to understand intuitively why this could be so.

I think here you got your $p$ and $q$ the wrong way around as we are using samples from $p$ to approximate our policy $q$. We typically will use importance sampling to generate samples from a different policy to our target policy for a few reasons - one reason might be that our target policy is hard to sample from whereas sampling from our behaviour policy $p$ might be relatively easy to sample from. Another reason is that we generally want to learn an optimal policy, but this could be difficult to learn if we don't explore enough. So we can follow some other policy that will explore sufficiently and still learn about our optimal target policy through the importance sampling ratio.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.