In example 5.5 of Sutton and Barto's book for proving infinite variance of first visit monte carlo ordinary importance sampling estimator, $\mathbb{E}[(\Pi_t\frac{\pi(A_t|S_t)}{b(A_t|S_t)}G_0)^2]$ is proven to be infinite by taking values from the example, but is there any way we can prove this in general without using any example?

  • $\begingroup$ The book says they prove that the IS scaled returns is infinite for that example only. IS samplings will be finite for tree like structures with finite starting states and no loops. In classical control theory such structure are called open loop control systems which are always stable (in the sense their output will not go to infinity). Whereas, a closed loop control system may or may not be stable. The example is pretty much analogous in the sense the self loop tries to reinforce the input i.e it scales input to a higher value and passes it to the initial stage, which is scaled again, so on. $\endgroup$ – DuttaA Sep 10 '20 at 3:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.