In slide 16 of his lecture 5 of the course "Reinforcement Learning", David Silver introduced GLIE Monte-Carlo Control.

enter image description here

But why is it an on-policy control? The sampling follows a policy $\pi$ while improvement follows an $\epsilon$-greedy policy, so isn't it an off-policy control?


In this case, $\pi$ has always been an $\epsilon$-greedy policy. In every iteration, this $\pi$ is used to generate ($\epsilon$-greedily) a trajectory from which the new $Q(s, a)$ values are calculated. The last line in the "pseudocode" tells you that the policy $\pi$ will be a new $\epsilon$-greedy policy in the next iteration. Since the policy that is improved and the policy that is sampled are the same, the learning method is considered an on-policy method.

If the last line was $\mu \leftarrow \epsilon\text{-greedy}(Q)$, it would be an off-policy method.

  • $\begingroup$ Not sure why you're using $\mu$ here while in the original post there's $\pi$, which is also the common letter to denote a policy (that is derived from $Q$), i.e. why did you revert my edit? $\endgroup$
    – nbro
    Nov 26 '20 at 19:20
  • $\begingroup$ @nbro Since the last line of the algorithm is $\pi\leftarrow \epsilon\text{-greedy}(Q)$, if I want to show an example for off-policy, I need to use another letter. I used $\mu$ because it is also used in David Silver's lecture slides (davidsilver.uk/wp-content/uploads/2020/03/control.pdf page5). Probably not the best example. Maybe a better way to make an example is to replace the first line with "Sample $k$th episode using $\mu$". (Edit: or maybe I could use $\pi'$ instead.) $\endgroup$
    – Hai Nguyen
    Nov 27 '20 at 18:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.