Derive Importance Sampling as Expected Value Notation

Question

I'm new to RL. Recently, I took a course on Coursera. In the Off-policy MC method, I learned the concept of Importance Sampling as follows:

where the importance sampling ratio is the ratio of the target policy over the behavior policy.

But in Suton book the expectation under the target policy is estimated like this:

Given that both sources used the same importance sampling ratio. However, I ended up getting $E[G_{t}|s] = \sum{G_{t} b \frac{\pi}{\pi}} = \sum{G_{t} \frac{1}{\rho}\pi} = E[\frac{G_{t}}{\rho_{t:T-1}}|s]$ instead

Did I do something wrong?

Answer 1

The importance sampling ratio is changing the measure of the expectation. We have the behaviour policy that generates trajectories in the environment, so we can calculate $\mathbb E_b[G_t|S_t = s_t] = v_b(s_t)$, but that's not what we want to calculate, we want to calculate $\mathbb E_\pi[G_t|S_t = s_t] = v_\pi(s_t)$. So, we need to find a scaling factor, $\rho$, that we can use in the first expectation to get to the second:

$$ \mathbb E_b[\rho G_t|S_t = s_t] = \mathbb E_\pi[G_t|S_t = s_t] = v_\pi(s_t) $$

That scaling factor is the importance sampling ratio. In your example, the first equation, $E[G_t|s]$, is something we aren't interested in because we don't want to improve the behaviour policy.