I had the same question when I am reading the RL textbook from Sutton Bartol as posted here.

enter image description here

Why do we update $W$ with $\frac{1}{\mu (A_t | S_t)}$ instead of $\frac{\pi (A_t | S_t)}{\mu (A_t | S_t)}$?

It seems that, with the updating rule from the textbook, whatever action $\mu$ decides to choose, we automatically assume that $\pi$ will choose it with 100% probability. But $\pi$ is greedy with respect to Q. How does this assumption make sense?

  • 2
    $\begingroup$ It says there that $\pi$ is a deterministic policy $\endgroup$ – Brale May 5 '20 at 17:43
  • 1
    $\begingroup$ Ah I see! It could be 0 but in case of 0, loop already breaks before updating W. Should have read the algorithms a few more times. Thank you very much! $\endgroup$ – roy May 5 '20 at 18:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.