This is a question from page 94 of Sutton and Barto's RL book 2020.

I read in someone's compiled GitHub answers to this book's exercises their answer was: "No because each state in an episode of blackjack is unique."

I think my answer is more yes, but I'm thinking in terms of casino blackjack, where they have multiple decks shuffled together and add in the dropped cards back into the deck every X games in order to prevent card counting and 1 game can be seen as an episode. I think in this case that first-visit MC and every-visit MC would have drastically different results, given that, at the start of the new episode, the state of the deck, which is only partially observed, will change the value of taking an action given a state (because I believe the cards left in the deck affect the value of an action, but the deck is not totally observable).

If this is blackjack, where the discarded cards are added back in and shuffled every episode, I'll agree that it shouldn't make a difference.

Are there any flaws in my conjecture?

  • $\begingroup$ Can you please provide the link to the specific Sutton and Barto's book that you're reading (given that there are many versions online)? Also, you say "someone's compiled GitHub answers to this book's exercises". Can you provide the link to the repo? $\endgroup$
    – nbro
    Dec 4 '20 at 13:40
  • $\begingroup$ I believe the question is here: incompleteideas.net/book/RLbook2020.pdf#page=94 The so-called GitHub solution is here: github.com/mharbuz/rlbook-exercises/blob/master/chapter5.md Now I have found an arguably more authoritative answer from here tams.informatik.uni-hamburg.de/lectures/2015ss/vorlesung/ML/… which reads "states never recur in the task, so no difference between first-visit and every-visit MC" $\endgroup$ Feb 27 at 3:05
  • 1
    $\begingroup$ In the book, Sutton and Barto, second edition (Reinforcement Learning: An Introduction) example 5.1 (Blackjack) in the second paragraph the author explicitly states that the version of Blackjack he is considering has in infinite deck with replacement - and so there is no utility to keeping track of the cards dealt. $\endgroup$ Nov 3 at 11:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.