1
$\begingroup$

Consider the following pseudocode of MC control with exploring starts:

enter image description here

When we choose $A_0$ randomly for state $S_0$, do we need to update the current $\pi(S_0)$ to the randomly chosen $A_0$ as well?

$\endgroup$

1 Answer 1

1
$\begingroup$

When we choose $A_0$ randomly for state $S_0$, do we need to update the current $\pi(S_0)$ to the randomly chosen $A_0$ as well?

No.

Under a given policy $\pi$, the action value of a state $Q_{\pi}(s,a)$ is the expected return when taking action $a$ in state $s$, and from that point on following the policy $\pi$. The action you are sampling for updates can be one that the policy would not take.

In general, when you are considering a specific action value - whether to sample it or update it - you are not concerned with how the agent got into the state and action involved. You only care what happens afterwards.

For exploring starts, you are not even concerned whether reaching any given state is possible under the policy being updated.

$\endgroup$
3
  • $\begingroup$ @ Neil Slater, in this code (which is the Matlab implementation, they do): "pol_pi(polInd) = unidrnd(2)-1; % FOR EXPLORING STARTS TAKE AN INITIAL RANDOM POLICY!!!" The link is this: waxworksmath.com/Authors/N_Z/Sutton/RLAI_1st_Edition/Code/… $\endgroup$ Commented Feb 16, 2022 at 21:15
  • $\begingroup$ @ Neil Slater, and what strange is: the code in the above link does not work if you do not do that update. However, with update it works perfectly! $\endgroup$ Commented Feb 16, 2022 at 21:18
  • 1
    $\begingroup$ @DSPinfinity I don't know enough Matlab to tell you why they do that, but on a quick look it doesn't seem to be an implmentation of exploring starts. Instead it simulates games from the start. That still works for the blackjack environment it seems. I wouldn't recommend you use that code to understand RL. $\endgroup$ Commented Feb 17, 2022 at 7:04

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .