4
$\begingroup$

In slide 16 of his lecture 5 of the course "Reinforcement Learning", David Silver introduced GLIE Monte-Carlo Control.

enter image description here

But why is it an on-policy control? The sampling follows a policy $\pi$ while improvement follows an $\epsilon$-greedy policy, so isn't it an off-policy control?

$\endgroup$

1 Answer 1

4
$\begingroup$

In this case, $\pi$ has always been an $\epsilon$-greedy policy. In every iteration, this $\pi$ is used to generate ($\epsilon$-greedily) a trajectory from which the new $Q(s, a)$ values are calculated. The last line in the "pseudocode" tells you that the policy $\pi$ will be a new $\epsilon$-greedy policy in the next iteration. Since the policy that is improved and the policy that is sampled are the same, the learning method is considered an on-policy method.

If the last line was $\mu \leftarrow \epsilon\text{-greedy}(Q)$, it would be an off-policy method.

$\endgroup$
2
  • $\begingroup$ Not sure why you're using $\mu$ here while in the original post there's $\pi$, which is also the common letter to denote a policy (that is derived from $Q$), i.e. why did you revert my edit? $\endgroup$
    – nbro
    Nov 26, 2020 at 19:20
  • $\begingroup$ @nbro Since the last line of the algorithm is $\pi\leftarrow \epsilon\text{-greedy}(Q)$, if I want to show an example for off-policy, I need to use another letter. I used $\mu$ because it is also used in David Silver's lecture slides (davidsilver.uk/wp-content/uploads/2020/03/control.pdf page5). Probably not the best example. Maybe a better way to make an example is to replace the first line with "Sample $k$th episode using $\mu$". (Edit: or maybe I could use $\pi'$ instead.) $\endgroup$
    – Hai Nguyen
    Nov 27, 2020 at 18:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .