I'm studying RL with Sutton and Barto's book. I'd like to ask about the order of execution of a statement in the algorithm below.

enter image description here

Here, $W$ (importance sampling ratio) is updated at the end of the For loop.

But, I think that updating should be located after calculating $G$ (the return) and before updating $C(s,a)$ (cumulative of $W$). This seems to be right considering the second picture below, which I found in http://incompleteideas.net/book/first/ebook/node56.html.

enter image description here

Is Sutton and Barto's book wrong? Or the two algorithms in Sutton and Barto's book and seconds picture are actually the same, and I am wrong? Is there any difference between the two when implemented? If I am wrong, can you explain the reason?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.