The average return for trajectories, $V^{\pi_e}$(s) is often computed via the importance sampling estimate $$V^{\pi_e}(s) = \frac{1}{n}\sum_{i=1}^n\prod_{t=0}^{H}\frac{\pi_e(a_t | s_t)}{\pi_b(a_t|s_t)}G_i$$ where $G_i$is the reward observed for the $i$th trajectory. Sutton and Barton gives an example whereby the variance could be infinite. In general however, why does this estimator suffer from high variance ? Is it because $\pi_e(a_t|s_t)$ is mainly deterministic and therefore, the importance weight is 0 for most trajectories, rendering those sample trajectories useless ?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.