5
$\begingroup$

Suppose there is an evaluation policy called $\pi_{e}$ and there are two behavior policies $\pi_{b1}$ and $\pi_{b2}$. I know that it is possible to estimate the return of policy $\pi_{e}$ through behavior policies via importance sampling, which is unbiased. But I do not know about the variance of return estimated through two behavior policies $\pi_{b1}$ and $\pi_{b2}$. Does anybody know about the variance or any bound on the variance of estimated return?

Let $G_{0}^{b1}=\sum_{t=1}^{T}\gamma^{t-1}r_{t}^{b1}$ represent the total return for an episode through behavior policy $\pi_{b1}$ and $G_{0}^{b2}=\sum_{t=1}^{T}\gamma^{t-1}r_{t}^{b2}$ represent the total return for an episode through behavior policy $\pi_{b2}$. It is possible to estimate the return of policy $\pi_{e}$ as follows:

$$G_{0}^{(e,b1)}=\prod_{t=1}^{T}\frac{\pi_{e}(a_{t}|s_{t})}{\pi_{b1}(a_{t}|s_{t})}*G_{0}^{b1}$$

$$G_{0}^{(e,b2)}=\prod_{t=1}^{T}\frac{\pi_{e}(a_{t}|s_{t})}{\pi_{b2}(a_{t}|s_{t})}*G_{0}^{b2}$$

I want to compare the variance of $G_{0}^{(e,b1)}$ and $G_{0}^{(e,b2)}$. Is there any formulation to compute the variance $G_{0}^{(e,b1)}$ and $G_{0}^{(e,b2)}$?

$\endgroup$
  • 1
    $\begingroup$ Perhaps give a formula or notes on a variance that you do know how to obtain in the question, so that someone answering can use the same notation, and understands where to start from so that you will understand $\endgroup$ – Neil Slater Jan 17 at 20:00
0
$\begingroup$

If you simulate many trajectories and receive many estimates of the two returns you're interested in, you could empirically compare their sample variances.

However, the variance of ordinary importance sampling is in general unbounded. If you're wanting some theoretical bounds on the variance of importance sampling estimates, I'd start with weighted importance sampling, whose variance converges to zero (Sutton and Barto, section 5.5).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.