4
$\begingroup$

I've been doing some research on the principles behind AlphaZero. Especially this cheat sheet (1) and this implementation (2) (in Connect 4) were very useful.

Yet, I still have two important questions:

  1. How is the policy network updated? In (2), board positions are saved in a dataset of tuples (state, policy, value). The value is derived from the result of the self-played game. However, I'm not sure which policy is saved: the number of times that each move has been played, the prior probabilities for each move (I guess not), or something else?

  2. The cheat sheet says that (for competitive play) the move is chosen with the greatest N (=most visited). Wouldn't it be more logical to choose the move with the highest probability calculated by the policy head?

Improve this question
$\endgroup$
1
$\begingroup$

However, I’m not sure which policy is saved

The policy from the Monte Carlo tree search is stored, as we can get the policy estimate from the network later by passing the given state through the network, which is used to calculate the cross entropy loss to update the network's policy (summed with Mean squared error loss between value head's prediction and the actual value/reward).

Wouldn’t it be more logical to choose the move with the highest probability calculated by the policy head?

Depends on the number of searches you've performed, after thousands of simulations the MCTS would give better results, as it approximates the minimax tree.

Improve this answer
$\endgroup$
3
  • $\begingroup$ To be more precise, I would say that "the visit counts" of the actions from the root noe of the Monte Carlo tree search are stored. Just "the policy" might be a bit too vague. Also, welcome to the site :) $\endgroup$
    – Dennis Soemers ♦
    Jul 29 '20 at 18:40
  • $\begingroup$ @DennisSoemers but aren't the visit counts of each action used to calculate the policy? $\endgroup$
    – Abhishek Kaushik
    Jul 30 '20 at 6:07
  • $\begingroup$ Sure, but "policy" can mean a lot of things. It can mean what you intend it to mean, you're not wrong. But the strategy we use in the "Selection" phase (like PUCT) is also often called a "tree policy". In Reinforcement Learning, a policy would be a mapping from states to a probability distribution over actions. You could view the visit counts (after normalising) as being such a (discrete) probability disrtibution.. but it could also refer to the policy network being trained! Hence, there's room for confusion. Explicitly saying that the visit counts gets stored leaves less room for confusion :) $\endgroup$
    – Dennis Soemers ♦
    Jul 30 '20 at 10:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.