# How can blackjack be formulated as a Markov decision process?

I am reading sutton barton's reinforcement learning textbook and have come across the finite Markov decision process (MDP) example of the blackjack game (Example 5.1).

Isn't the environment constantly changing in this game? How would the transition probabilities be fixed in such an environment, when both you and the dealer draw cards?

Isn't the environment constantly changing in this game?

The current state of the agent and the environment is constantly changing as you play, but not necessarily the transition probabilities. For simplicity, you may assume that the transition probabilities do not change (e.g. if the dealer and the deck are the same every time you play).

How would the transition probabilities be fixed in such an environment, when both you and the dealer draw cards?

The actions of the dealer will be incorporated into the transition probabilities of the environment. Whenever the player (or RL agent) takes an action, then it will receive a reward, according to the reward function of the environment (the rules of the game), and the agent and the environment will be moved to the next state, according to the transition probabilities, which do not have to change for stochasticity to occur in the environment. In fact, these transition probabilities already incorporate this stochasticity.

Also, if even the environment was changing, you could still model your problem as an MDP, but the MDP would change accordingly.

The example 5.1 of the book (that you mention) actually explains in detail how to formulate this game as a finite MDP.

• In fact the book ignores drawn cards and (unrealistically) assumes drawing with replacement, because it is simpler to formulate. Drawing without replacemant still results in a valid MDP as you suggest, but the state model becomes far more complex for little gain when used as a teaching example – Neil Slater May 9 at 12:01