In Chapter 1 of the book Reinformcement Learning An Introduction 2nd Edition by Richard S. Sutton and Andrew G.Barto, there is one statement "Exploratory moves do not result in any learning".

This sentence is in Figure 1.1.

Figure 1.1: A sequence of tic-tac-toe moves. The solid black lines represent the moves taken during a game; the dashed lines represent moves that we (our reinforcement learning player) considered but did not make. Our second move was an exploratory move, meaning that it was taken even though another sibling move, the one leading to e⇤, was ranked higher. Exploratory moves do not result in any learning, but each of our other moves does, causing updates as suggested by the red arrows in which estimated values are moved up the tree from later nodes to earlier nodes as detailed in the text.

It confuses me. In my understanding, exploration should contribute to learning in almost all RL algorithms. So, why does the book state "Exploratory moves do not result in any learning" in this case?


I believe this is a pedagogical decision. Because the sentence occurs in the first chapter of the book, I think the authors are trying to avoid the objection that a neophyte might make: learning from random movements seems like it will cause you to learn strange behaviors.

Certainly, the statement is inaccurate. We need only reach page 26 to see a counterexample: The Q-learning equation (2.1) sums over all actions, not just the greedy ones. This also applies to the temporal difference learning method specified in Chapter 6, which is the method Figure 1.1 is discussing.

| improve this answer | |
  • 2
    $\begingroup$ I would say that the statement is inaccurate rather than incorrect. In off-policy learning with current greedy policy as target policy, exploratory actions are not backed up to cause value estimate updates (or if you prefer, the importance sampling weighting is zero), whilst greedy moves are. When I read the book, I took the statement in that sense. $\endgroup$ – Neil Slater Oct 25 '19 at 6:55
  • $\begingroup$ @NeilSlater that's a good point. I'll update the answer. $\endgroup$ – John Doucette Oct 25 '19 at 11:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.