1
$\begingroup$

I'm studying RL with Sutton and Barto's book. I'd like to ask about the order of execution of a statement in the algorithm below.

enter image description here

Here, $W$ (importance sampling ratio) is updated at the end of the For loop.

But, I think that updating should be located after calculating $G$ (the return) and before updating $C(s,a)$ (cumulative of $W$). This seems to be right considering the second picture below, which I found in http://incompleteideas.net/book/first/ebook/node56.html.

enter image description here

Is Sutton and Barto's book wrong? Or the two algorithms in Sutton and Barto's book and seconds picture are actually the same, and I am wrong? Is there any difference between the two when implemented? If I am wrong, can you explain the reason?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.