What is a policy training target in AlphaZero?

Question

In AlphaZero's attached pseudocode, they create a training target for the policy network in this way.

def store_search_statistics(self, root):
    sum_visits = sum(child.visit_count for child in root.children.itervalues())
    self.child_visits.append([
        root.children[a].visit_count / sum_visits if a in root.children else 0
        for a in range(self.num_actions)
    ])

In other words, the training target probability for a certain move is proportional to its visit count.

However, in the paper, they describe the usage of softmax sampling of visit counts with temperature. This temperate is equal to 1 for the first 30 moves (in this case the policy training target is the same as in the pseudocode above) and for subsequent moves they set infinitesimal temperature -> 0, which essentially means they are picking the move with the highest visit count.

Since these are 2 different things (if the game has more than 30 moves), my question is: which approach should be used for creating the training target for the policy?

Answer 1

The training target for the policy is always exactly the one described by the pseudocode; distribution proportional to visit counts, without any other kind of scaling.

The softmax sampling with the temperature (different for first 30 moves than after that) is not used for the policy target, but is used for the "real" move selection by the agent in the self-play game (i.e. after it has run its search of 800 or 1600 or however many iterations it was for the root game state encountered in the self-play game).

---

Generally, in my experience, you really wouldn't want the policy target to become too close to deterministic, except if really your entire MCTS already is extremely convinced about one move being clearly the best one all by itself (without getting pushed towards something more deterministic by a low-temperature softmax). If your policy becomes too deterministic too quickly, it destroys the exploration that we normally want MCTS to do inside its selection phase. Using a low-temperature softmax for the policy training target would introduce quite a big risk of this happening.