1
$\begingroup$

enter image description here

Why is importance sampling ratio in n-step TD is multiplying error rather than return? In Monte Carlo methods for state values, importance sampling ratio was simply a multiplier for the return.

$\endgroup$
1
  • $\begingroup$ can you please post to the part of the book stating the MC update reweighting as you are reporting? $\endgroup$
    – Alberto
    Commented May 13 at 11:05

1 Answer 1

0
$\begingroup$

No, also for MC Control the importance sampling weights the update, and not just the return

enter image description here

But the reason I think can be understood intuitively as:

  • if you have the same policy as the behavioral, then the updates should be the same, thus the ratio is 1
  • if the behavioral policy takes an action that you would have never taken, then the update should be 0 (as you would never go to that part of the MDP), thus the ratio is 0
$\endgroup$
1
  • $\begingroup$ I updated the question: In Monte Carlo methods for state values, importance sampling ratio was simply a multiplier for the return. $\endgroup$ Commented May 12 at 23:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .