2
$\begingroup$

I am familiar with the Monte Carlo Tree Search, which can be broken down into 4 parts - Selection, Expansion, Simulation and Backpropogation. In this case, the simulation step denotes performing a random rollout from the expanded node. From my understanding, the AlphaGoZero paper removes the MCTS simulation step and replaces it with a neural network that takes the leaf node as input and outputs a policy vector and a value.

However, I am not sure what the AlphaGoZero paper means when they use the term simulation. It contains the following paragraph - "The MCTS uses the neural network fθ to guide its simulations (see Fig. 2). Each edge (s, a) in the search tree stores a prior probability P(s, a), a visit count N(s, a), and an action value Q(s, a). Each simulation starts from the root state and iteratively selects moves that maximize an upper confidence bound Q(s, a)+U(s, a), where U(s, a)∝P(s, a)/(1+N(s, a)) (refs 12, 24), until a leaf node s′ is encountered."

$\endgroup$

1 Answer 1

2
$\begingroup$

Your observation is accurate. The use of the term "simulation" in the AlphaGoZero paper can indeed be confusing. It does not refer to the simulation or rollout phase of the Monte Carlo Tree Search. As you correctly pointed out, the AlphaGoZero model removes the rollout, which in the AlphaGo paper was a shallow neural network trained on supervised data. Instead, it uses a single neural network that outputs both policies and values for the given position in the expansion phase.

In the excerpt you have provided, the term "simulation" refers to the exploration and play within the game tree. I would normally refer to the simulation phase as "rollout" to avoid confusion, but that is up to you. Otherwise, your understanding is correct.

I hope this clarifies your doubts. If I have not made myself clear, please let me know.

$\endgroup$
2
  • $\begingroup$ Thank you for your reply. I am a little unsure of the word "exploration" here. Why do you think it's just exploration that's going on during simulation? $\endgroup$ Aug 3, 2023 at 16:29
  • 1
    $\begingroup$ When I say "exploration," I am referring to the fact that the tree is increasing in size because it is exploring more nodes in the game. Reinforcement learning (RL) has the concept of exploration vs. exploitation, which comes from the idea of balancing between repeating what seems to be good actions and exploring new states and actions that have not been explored before. The MCTS algorithm is constructed in a way to make this balance efficient for an RL agent. However, when I said "exploration," I was referring to executing an MCTS iteration and expanding the tree. $\endgroup$
    – Cesar Ruiz
    Aug 3, 2023 at 20:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .