0
$\begingroup$

I'm new to RL. Recently, I took a course on Coursera. In the Off-policy MC method, I learned the concept of Importance Sampling as follows:

enter image description here

where the importance sampling ratio is the ratio of the target policy over the behavior policy.

But in Suton book the expectation under the target policy is estimated like this:

enter image description here

Given that both sources used the same importance sampling ratio. However, I ended up getting $E[G_{t}|s] = \sum{G_{t} b \frac{\pi}{\pi}} = \sum{G_{t} \frac{1}{\rho}\pi} = E[\frac{G_{t}}{\rho_{t:T-1}}|s]$ instead

Did I do something wrong?

$\endgroup$
0
1
$\begingroup$

The importance sampling ratio is changing the measure of the expectation. We have the behaviour policy that generates trajectories in the environment, so we can calculate $\mathbb E_b[G_t|S_t = s_t] = v_b(s_t)$, but that's not what we want to calculate, we want to calculate $\mathbb E_\pi[G_t|S_t = s_t] = v_\pi(s_t)$. So, we need to find a scaling factor, $\rho$, that we can use in the first expectation to get to the second:

$$ \mathbb E_b[\rho G_t|S_t = s_t] = \mathbb E_\pi[G_t|S_t = s_t] = v_\pi(s_t) $$

That scaling factor is the importance sampling ratio. In your example, the first equation, $E[G_t|s]$, is something we aren't interested in because we don't want to improve the behaviour policy.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.