I've been doing some research on the principles behind AlphaZero. Especially this cheat sheet (1) and this implementation (2) (in Connect 4) were very useful.

Yet, I still have two important questions:

  1. How is the policy network updated? In (2), board positions are saved in a dataset of tuples (state, policy, value). The value is derived from the result of the self-played game. However, I'm not sure which policy is saved: the number of times that each move has been played, the prior probabilities for each move (I guess not), or something else?

  2. The cheat sheet says that (for competitive play) the move is chosen with the greatest N (=most visited). Wouldn't it be more logical to choose the move with the highest probability calculated by the policy head?


1 Answer 1


However, I’m not sure which policy is saved

The policy from the Monte Carlo tree search is stored, as we can get the policy estimate from the network later by passing the given state through the network, which is used to calculate the cross entropy loss to update the network's policy (summed with Mean squared error loss between value head's prediction and the actual value/reward).

Wouldn’t it be more logical to choose the move with the highest probability calculated by the policy head?

Depends on the number of searches you've performed, after thousands of simulations the MCTS would give better results, as it approximates the minimax tree.

  • $\begingroup$ To be more precise, I would say that "the visit counts" of the actions from the root noe of the Monte Carlo tree search are stored. Just "the policy" might be a bit too vague. Also, welcome to the site :) $\endgroup$
    – Dennis Soemers
    Commented Jul 29, 2020 at 18:40
  • $\begingroup$ @DennisSoemers but aren't the visit counts of each action used to calculate the policy? $\endgroup$ Commented Jul 30, 2020 at 6:07
  • $\begingroup$ Sure, but "policy" can mean a lot of things. It can mean what you intend it to mean, you're not wrong. But the strategy we use in the "Selection" phase (like PUCT) is also often called a "tree policy". In Reinforcement Learning, a policy would be a mapping from states to a probability distribution over actions. You could view the visit counts (after normalising) as being such a (discrete) probability disrtibution.. but it could also refer to the policy network being trained! Hence, there's room for confusion. Explicitly saying that the visit counts gets stored leaves less room for confusion :) $\endgroup$
    – Dennis Soemers
    Commented Jul 30, 2020 at 10:02

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .