2
$\begingroup$

In Sutton and Barto on chapter 5 (p.96), they talk about estimating state-action values with Monte Carlo:

For policy evaluation to work for action values, we must assure continual exploration. One way to do this is by specifying that the episodes start in a state–action pair, and that every pair has a nonzero probability of being selected as the start. This guarantees that all state–action pairs will be visited an infinite number of times in the limit of an infinite number of episodes. We call this the assumption of exploring starts.

What does it mean for an episode to start in a state-action pair? Examples are welcome

$\endgroup$

1 Answer 1

2
$\begingroup$

It simply means that you start an episode in any possible state-action combination. Consider a gridworld type environment: an example of starting in a state-action pair would be to start in the top left most square with the action selected as being 'down'. If you do this random initialisation for each episode then in the limit of infinite episodes you will see each possible state-action pair infinitely often, ensuring sufficient exploration.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .