The average return for trajectories, $V^{\pi_e}$(s) is often computed via the importance sampling estimate $$V^{\pi_e}(s) = \frac{1}{n}\sum_{i=1}^n\prod_{t=0}^{H}\frac{\pi_e(a_t | s_t)}{\pi_b(a_t|s_t)}G_i$$ where $G_i$is the reward observed for the $i$th trajectory. Sutton and Barton gives an example whereby the variance could be infinite.

In general, however, why does this estimator suffer from high variance? Is it because $\pi_e(a_t|s_t)$ is mainly deterministic and, therefore, the importance weight is $0$ for most trajectories, rendering those sample trajectories useless?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.