Importance sampling is a common method for calculating off-policy estimates in RL. I have been reading through some of the original documentation (D.G. Horvitz and D.J. Thompson, Powell, M.J. and Swann, J) and cannot find any restrictions on the reward or value being estimated. However, it seems that there are constraints because the calculation is not what I would expect for RL environments that have negative rewards.

For example, consider for a given action-state pair ($a_i, s_i$), $\pi_e(a|s) = 0.4$ and $\pi_b(a|s) = 0.6,$ where $\pi_b$ and $\pi_e$ are the behavioral and evaluation policies respectively. Also, assume the reward range is $[-1,0]$, and this action has a reward of $r_{\pi_b}=-0.5$.

Under the IS definition, the expected reward under $\pi_b$ would be $r_{\pi_e} = \frac{\pi_b(a|s)}{\pi_e(a|s)} r_{\pi_b}$. In this example, $r_{\pi_e}=-0.75$ thus $r_{\pi_e} < r_{\pi_b}$. However, assuming a change of scale of the reward to be $[0,1]$ which result in $r_{\pi_b}=0.5$, results in $r_{\pi_e} > r_{\pi_b}$.

All examples of IS I have seen in reference focus on positive rewards. However, I find myself wondering if this formulation applies to negative rewards too. If this formulation does allow for negative reward structures, I'm not sure how to interpret this result. I'm wondering how changing the scale of the reward could change the order? Is there any documentation on the requirements of the value in IS? Any insight into this would be greatly appreciated!

  • $\begingroup$ When you say "original documentation (D.G. Horvitz and D.J. Thompson, Powell, M.J. and Swann, J)", are you referring to a paper? If yes, can you provide the link? One of the papers you are referring to may be A Generalization of Sampling Without Replacement from a Finite Universe. Is this correct? What about the other? $\endgroup$
    – nbro
    Nov 5 '20 at 22:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.