According to a blog post by DeepMind, AlphaZero doesn't have a real rollout.

AlphaGo Zero does not use "rollouts" - fast, random games used by other Go programs to predict which player will win from the current board position. Instead, it relies on its high quality neural networks to evaluate positions.

Instead, I assume it just interprets the winner at a given state by the NN values head result. This replaces the rollout. So the computation time saved could be used for many expansions instead. Evaluating a state from a root node would then be the best action derived from the visit count in MCTS, which is only based on the predictions of the NN value heads. (no current score, no policy?)

With policy, I mean the NN's policy head (softmax).

This would mean that the NN policy is only used in the loss calculation and nowhere else?

  • $\begingroup$ You have lot's of questions and interpretations there. So I'll do my best to explain it below. $\endgroup$ Aug 21 at 6:00

TL;DR: Alpha Zero removed rollout altogether from MCTS and just used current DNN estimates instead.

The single Deep Neural Network has 2 heads:

  • A Value Head (which assigns a score to each state).
  • And a Policy Head (which predicts the score for all possible moves).

Instead of doing rollouts to determine the outcome, it uses the DNN estimations, so it don't need to explore too deep.

By relying on the DNN, the MCTS gets even simpler:

  • Probability of each action is a simple normalization of previously obtained Policy Head.
  • Selection chooses move with “low count, high move probability, and high value”
  • Expansion is done by DNN, outputing Value and Policy.
  • Simulation with rollouts: no longer needed.
  • Back-propagation updates nodes using DNN's value (instead of a rollout outcome).



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.