This is a question from page 94 of Sutton and Barto's RL book 2020.

I read in someone's compiled GitHub answers to this book's exercises their answer was: "No because each state in an episode of blackjack is unique."

I think my answer is more yes, but I'm thinking in terms of casino blackjack, where they have multiple decks shuffled together and add in the dropped cards back into the deck every X games in order to prevent card counting and 1 game can be seen as an episode. I think in this case that first-visit MC and every-visit MC would have drastically different results, given that, at the start of the new episode, the state of the deck, which is only partially observed, will change the value of taking an action given a state (because I believe the cards left in the deck affect the value of an action, but the deck is not totally observable).

If this is blackjack, where the discarded cards are added back in and shuffled every episode, I'll agree that it shouldn't make a difference.

Are there any flaws in my conjecture?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.