I'm attempting to create a Tic Tac Toe player using MCTS. For the game environment, I'm using Tic Tac Toe from the Gymnasium/Petting Zoo environment.

Running MCTS on Tic Tac Toe requires simulating hypothetical board states. To do this, my idea is to copy the state of the board at the root node, creating a new environment where I could simulate a hypothetical move.

I don't see any built in functions for this, so I attempted to see if copy.deepcopy() would copy the board state for me:

import copy
from pettingzoo.classic import tictactoe_v3

env = tictactoe_v3.env(render_mode=None)

env = copy.deepcopy(env)

This gives an error: AttributeError: 'raw_env' object has no attribute '_cumulative_rewards'

Clearly, deepcopy is not fully copying the board state. So how am I supposed to consider hypothetical board positions starting from a given state? Technically I could replay every action in the history to get the new board there, but that seems wildly inefficient, especially since I plan to play more complex games in the gym environment after this.


1 Answer 1


So how am I supposed to consider hypothetical board positions starting from a given state?

In an environment designed for model-free reinforcement learning, you cannot, at least not via any of the public functions. That's because look-ahead search is not really a part of standard RL toolkit.

If you want to run a model-based algorithm for tree search algorithms, you need a separate game model implemented. One model instance is maintaining state for the environment, and the second model instance can be used for planning. Sometimes gym libraries do decompose into an environment wrapper and a game engine that you could do this with, but that needs to be checked per environment and there is no standard approach here. The only rule is that the standard gym environment interface has to exist (e.g. reset and step functions).

A lot of the simpler environments directly implement game rules as functions within a single environment class, and cannot be separated out. This is the case with tictactoe_v3 (the source for the step method is a long function with significant game rules implemented, not available separately) - in which case the normal option is to implement a new game model and use that for search/planning algorithms.

  • $\begingroup$ So you need to build a TicTacToe class from scratch in this case? Why use Gymnasium at all, if you have to build a model of the game from scratch (said game model could trivially implement a step() function, so why use Gymnasium)? Also, will this be the case for Connect Four and Chess? This seems incredibly inconvenient, I thought Gymnasium was intended to be agnostic to which reinforcement learning approach you were taking. $\endgroup$
    – Ben G
    Feb 24 at 0:52
  • $\begingroup$ Note that I am attempting to implement the AlphaZero approach to TicTacToe, and later to Connect Four and Chess. I'm amazed that the most prominent RL environment would not support one of the most prominent approaches to RL in recent history. $\endgroup$
    – Ben G
    Feb 24 at 1:00
  • $\begingroup$ @BenG it is not as inconvenient as you seem to think. You will still spend far longer on your RL implementation. Either way, I didn't design or write this library. Don't shoot the messenger, and I hope you find the answer useful as it is factual $\endgroup$ Feb 24 at 11:41
  • $\begingroup$ Gym is not very useful for two player games in any case, as it doesn't really support multiple agent training. When I implemented a Connect 4 agent, I just wrote the model. For chess, maybe look for an engine. $\endgroup$ Feb 24 at 11:45
  • $\begingroup$ The dissapointing thing is that I was looking forward to a standardized environment for playing these games, with a common interface. So that the same system could play multiple games. But it seems like no one has created such a common interface and you need to implement each game individually $\endgroup$
    – Ben G
    Feb 24 at 18:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .