3
$\begingroup$

In a vanilla Monte Carlo tree search (MCTS) implementation, the rollout is usually implemented following a uniform random policy, that is, it takes random actions until the game is finished and only then the information gathered is backed up.

I have read the AlphaZero paper (and the AlphaGo Zero too) and I didn't find any information on how the rollout is implemented (maybe I missed it).

How is the rollout from the MCTS implemented in both of the AlphaGo Zero and the AlphaZero algorithms?

$\endgroup$
  • 1
    $\begingroup$ AlphaZero/Go don't have rollout. Rollout replaced with neural network value estimation $\endgroup$ – mirror2image Nov 3 at 7:40
  • $\begingroup$ @mi: I thought that original Alpha Go did have a rollout policy, but that one of the simplifying changes in AlphaZero was to completely remove it? $\endgroup$ – Neil Slater Nov 3 at 9:03
  • 1
    $\begingroup$ @NeilSlater That's correct. The question seems to be about AlphaGo Zero and AlphaZero though, not about Alpha Go :) AlphaGo: uses a fast rollout policy (trained like the policy network, but is not a large DNN, just a single layer + softmax I believe). AlphaGo Zero: no rollouts. AlphaZero: no rollouts. $\endgroup$ – Dennis Soemers Nov 3 at 13:53
  • $\begingroup$ @mirror2image even though it was only a line, you answered the question -- why not make it slightly more verbose and make it an answer $\endgroup$ – mshlis Nov 3 at 17:57
  • $\begingroup$ Thanks for the answers. However, I'm still confused. I understand that they use the V value returned from the network to simulate who won, but how do they know which value of V a player won or lost? Because as far as I understand, in the weight update (in the loss function), they will compare the probability distribution gathered from the self-plays and the difference between the real winner (from MCTS) and the value V. $\endgroup$ – ihavenoidea Nov 3 at 23:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.