Consider the following pseudocode of MC control with exploring starts:
When we choose $A_0$ randomly for state $S_0$, do we need to update the current $\pi(S_0)$ to the randomly chosen $A_0$ as well?
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
When we choose $A_0$ randomly for state $S_0$, do we need to update the current $\pi(S_0)$ to the randomly chosen $A_0$ as well?
No.
Under a given policy $\pi$, the action value of a state $Q_{\pi}(s,a)$ is the expected return when taking action $a$ in state $s$, and from that point on following the policy $\pi$. The action you are sampling for updates can be one that the policy would not take.
In general, when you are considering a specific action value - whether to sample it or update it - you are not concerned with how the agent got into the state and action involved. You only care what happens afterwards.
For exploring starts, you are not even concerned whether reaching any given state is possible under the policy being updated.
answered Feb 16, 2022 at 21:01
-
$\begingroup$ @ Neil Slater, in this code (which is the Matlab implementation, they do): "pol_pi(polInd) = unidrnd(2)-1; % FOR EXPLORING STARTS TAKE AN INITIAL RANDOM POLICY!!!" The link is this: waxworksmath.com/Authors/N_Z/Sutton/RLAI_1st_Edition/Code/… $\endgroup$ Feb 16, 2022 at 21:15
-
$\begingroup$ @ Neil Slater, and what strange is: the code in the above link does not work if you do not do that update. However, with update it works perfectly! $\endgroup$ Feb 16, 2022 at 21:18
-
1$\begingroup$ @DSPinfinity I don't know enough Matlab to tell you why they do that, but on a quick look it doesn't seem to be an implmentation of exploring starts. Instead it simulates games from the start. That still works for the blackjack environment it seems. I wouldn't recommend you use that code to understand RL. $\endgroup$ Feb 17, 2022 at 7:04