How do I compute the variance of the return of an evaluation policy using two behaviour policies?

Question

Suppose there is an evaluation policy called $\pi_{e}$ and there are two behavior policies $\pi_{b1}$ and $\pi_{b2}$. I know that it is possible to estimate the return of policy $\pi_{e}$ through behavior policies via importance sampling, which is unbiased. But I do not know about the variance of return estimated through two behavior policies $\pi_{b1}$ and $\pi_{b2}$. Does anybody know about the variance or any bound on the variance of estimated return?

Let $G_{0}^{b1}=\sum_{t=1}^{T}\gamma^{t-1}r_{t}^{b1}$ represent the total return for an episode through behavior policy $\pi_{b1}$ and $G_{0}^{b2}=\sum_{t=1}^{T}\gamma^{t-1}r_{t}^{b2}$ represent the total return for an episode through behavior policy $\pi_{b2}$. It is possible to estimate the return of policy $\pi_{e}$ as follows:

$$G_{0}^{(e,b1)}=\prod_{t=1}^{T}\frac{\pi_{e}(a_{t}|s_{t})}{\pi_{b1}(a_{t}|s_{t})}*G_{0}^{b1}$$

$$G_{0}^{(e,b2)}=\prod_{t=1}^{T}\frac{\pi_{e}(a_{t}|s_{t})}{\pi_{b2}(a_{t}|s_{t})}*G_{0}^{b2}$$

I want to compare the variance of $G_{0}^{(e,b1)}$ and $G_{0}^{(e,b2)}$. Is there any formulation to compute the variance $G_{0}^{(e,b1)}$ and $G_{0}^{(e,b2)}$?

Answer 1

If you simulate many trajectories and receive many estimates of the two returns you're interested in, you could empirically compare their sample variances.

However, the variance of ordinary importance sampling is in general unbounded. If you're wanting some theoretical bounds on the variance of importance sampling estimates, I'd start with weighted importance sampling, whose variance converges to zero (Sutton and Barto, section 5.5).