3
$\begingroup$

In Chapter 1 of the book Reinformcement Learning An Introduction 2nd Edition by Richard S. Sutton and Andrew G.Barto, there is one statement "Exploratory moves do not result in any learning".

This sentence is in Figure 1.1.

Figure 1.1: A sequence of tic-tac-toe moves. The solid black lines represent the moves taken during a game; the dashed lines represent moves that we (our reinforcement learning player) considered but did not make. Our second move was an exploratory move, meaning that it was taken even though another sibling move, the one leading to e⇤, was ranked higher. Exploratory moves do not result in any learning, but each of our other moves does, causing updates as suggested by the red arrows in which estimated values are moved up the tree from later nodes to earlier nodes as detailed in the text.

It confuses me. In my understanding, exploration should contribute to learning in almost all RL algorithms. So, why does the book state "Exploratory moves do not result in any learning" in this case?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

I believe this is a pedagogical decision. Because the sentence occurs in the first chapter of the book, I think the authors are trying to avoid the objection that a neophyte might make: learning from random movements seems like it will cause you to learn strange behaviors.

Certainly, the statement is inaccurate. We need only reach page 26 to see a counterexample: The Q-learning equation (2.1) sums over all actions, not just the greedy ones. This also applies to the temporal difference learning method specified in Chapter 6, which is the method Figure 1.1 is discussing.

Improve this answer
$\endgroup$
2
  • 2
    $\begingroup$ I would say that the statement is inaccurate rather than incorrect. In off-policy learning with current greedy policy as target policy, exploratory actions are not backed up to cause value estimate updates (or if you prefer, the importance sampling weighting is zero), whilst greedy moves are. When I read the book, I took the statement in that sense. $\endgroup$
    – Neil Slater
    Oct 25, 2019 at 6:55
  • $\begingroup$ @NeilSlater that's a good point. I'll update the answer. $\endgroup$
    – John Doucette
    Oct 25, 2019 at 11:46

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .