The formula in question uses a function N(state, action) that defines a visit count of a state-action pair (introduced on page 3). To describe how it is used, lets first describe the steps of AlphaGo Zero as a whole.
There are 4 "phases" to the Monte-Carlo tree search in AlphaGo Zero as depicted in Figure 2. The first 3 expand and update the tree and together are the "search" in Monte-Carlo tree "search" in AlphaGo Zero.
Select an edge (action) in the tree with maximum action-value Q (plus upper confidence bound U)
Expand and evaluate the leaf node using the network
Backup Action-values Q are updated to track the evaluations of the Value in the subtree.
Play - After the "search" is complete, the search probabilities are returned proportional to the visitation count of the nodes of the tree.*
* This is where the formula in question comes into play (pun intended). During the "search", nodes that looked good were expanded on and thus their visitation counts were updated. So the formula is essentially describing this logic:
Good nodes have higher counts, so choose the nodes with higher counts often
But what if there is a really good node but it hasn't been visited a lot so its not chosen?
This is where the temperature parameter comes in:
- If the temperature is 1, this selects moves proportionally to their visit counts.
- If the temperature is 0 (not actually but rather in infinitesimal that approaches 0), and with some added noise, it "ensures that all moves may be tried, but the search may still overrule bad moves."
So altogether, the formula is saying : Pick good things most of the time
The mathematical evaluation of the formula is described below:
AlphaGo Zero defines the probability of each action (aka the policy) by that formula. If there are 3 nodes, A, B and C, and they have each been visited 10, 70, and 20 times (100 times in total), respectively, then the probability of taking those actions is:
- P(A) = 10/100 = .10
- P(B) = 70/100 = .7
- P(C) = 20/100 = .2