I've been doing some research on the principles behind AlphaZero. Especially this cheat sheet (1) and this implementation (2) (in Connect 4) were very useful.
Yet, I still have two important questions:
How is the policy network updated? In (2), board positions are saved in a dataset of tuples (state, policy, value). The value is derived from the result of the self-played game. However, I'm not sure which policy is saved: the number of times that each move has been played, the prior probabilities for each move (I guess not), or something else?
The cheat sheet says that (for competitive play) the move is chosen with the greatest N (=most visited). Wouldn't it be more logical to choose the move with the highest probability calculated by the policy head?