# Why is GLIE Monte-Carlo control an on-policy control?

In slide 16 of his lecture 5 of the course "Reinforcement Learning", David Silver introduced GLIE Monte-Carlo Control.

But why is it an on-policy control? The sampling follows a policy $$\pi$$ while improvement follows an $$\epsilon$$-greedy policy, so isn't it an off-policy control?

In this case, $$\pi$$ has always been an $$\epsilon$$-greedy policy. In every iteration, this $$\pi$$ is used to generate ($$\epsilon$$-greedily) a trajectory from which the new $$Q(s, a)$$ values are calculated. The last line in the "pseudocode" tells you that the policy $$\pi$$ will be a new $$\epsilon$$-greedy policy in the next iteration. Since the policy that is improved and the policy that is sampled are the same, the learning method is considered an on-policy method.
If the last line was $$\mu \leftarrow \epsilon\text{-greedy}(Q)$$, it would be an off-policy method.
• Not sure why you're using $\mu$ here while in the original post there's $\pi$, which is also the common letter to denote a policy (that is derived from $Q$), i.e. why did you revert my edit? – nbro Nov 26 '20 at 19:20
• @nbro Since the last line of the algorithm is $\pi\leftarrow \epsilon\text{-greedy}(Q)$, if I want to show an example for off-policy, I need to use another letter. I used $\mu$ because it is also used in David Silver's lecture slides (davidsilver.uk/wp-content/uploads/2020/03/control.pdf page5). Probably not the best example. Maybe a better way to make an example is to replace the first line with "Sample $k$th episode using $\mu$". (Edit: or maybe I could use $\pi'$ instead.) – Hai Nguyen Nov 27 '20 at 18:25