If AlphaZero was always playing the best moves it would just generate the same training game over and over again. So where does the randomness come from? When does it decide not to play the most optimal move?
$\begingroup$
$\endgroup$
2
-
$\begingroup$ AlphaZero does not know the optimal move. It has not "solved" the games of go or of chess etc. So could you clarify that you are looking for how the AlphaZero algorithm explores state/action space during training only (technically it explores during non-training play as well)? $\endgroup$– Neil SlaterAug 27, 2019 at 21:02
-
$\begingroup$ The optimal move according to the network. By playing the moves it thinks are best it would just play the same moves. For example it plays first move e4 because it is the best move then it sees the new position and thinks that e5 is the best for black and it would just repeat this singe game over and over without producing any meaningful training data $\endgroup$– Bojidar IvanovAug 27, 2019 at 22:20
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
During the self-play training process, AlphaZero does not greedily play only the moves it thinks are "best" (which would normally be the move with the highest visit count leading out of the root node of the MCTS search tree). Instead, for the purpose of generating a more diverse set of experience, it samples moves proportionally to the visit counts. This means that in any given situation encountered during self-play, the move considered to be "optimal" will still have the largest probability of getting picked, but other moves will also have smaller probabilities of getting picked. In theory, it might even sometimes pick the move that it expects to be the worst one (very rarely)!
If I recall correctly, they only did what I describe above for the first 30 moves of any game, and afterwards move on to greedy play. This still results in a very diverse set of 30-move-starts for every game it experiences though. I'm not 100% sure if I remember this detail correctly though, maybe they only did this earlier (in AlphaGo Zero for example), and no longer do it in AlphaZero. Would have to check the paper to make sure.
Additionally, whenever they start a new search process, they perturb the prior probabilities assigned by the learned policy network to all the moves available in the root node. This is done in a non-deterministic way using Dirichlet noise. I think this is not really explicitly mentioned in the AlphaZero paper, but it is in the Supplementary Materials (and also in the AlphaGo Zero paper?). Anyway, this also means that if precisely the same game state is encountered twice in two different games of self-play, the search behaviour may be slightly different due to the introduced stochasticity, and hence it may come to a different conclusion as to what move is "optimal".
answered Aug 28, 2019 at 8:49