GLIE+MC control Algorithm:
My question is why does this algorithm use only a single Monte Carlo episode (during PE step) to compute the $Q(s,a)$? In my understanding this has the following drawbacks:
- If we have multiple terminal states then we will only reach one (per Policy Iteration step PE+PI).
- It is highly unlikely that we will visit all the states (during training), and a popular scheduling algorithm for exploration constant $\epsilon = 1/k$ where $k$ is apparently the episode number, ensures that exploration decays very very rapidly. This ensures that we may never visit a state during our entire training.
So why this algorithm uses single MC episode and why not multiple episodes in a single Policy Iteration step so that the agent gets a better feel of the environment?