At what point are MCTS results discarded in AlphaZero Training?

Question

Regarding the AlphaZero paper, it is not clear to me when the Monte Carlo Tree Search (MCTS) results will be cleaned up.

I assume this has to happen at some point, since mixing results could lead to lower quality results? Imagine in the self-play the Neural Network (NN) is updated to a new version and evaluates certain patterns differently by detecting a new trick. Many iterations must follow to outperform the old best choice (visit-count). I imagine discarding old MCTS results should be done about between an episode and the next NN weight updates.

I feel that a wrong decision here could have a strong negative impact on the overall learning process.

Answer 1

You're right, it would not be great to keep training on games played early during the selfplay process, since they would just hurt the network playing strength.

The AlphaZero paper (free access) itself does not elaborate, but it has this sneaky sentence:

Unless otherwise specified, the training and search algorithm and parameters are identical to AlphaGo Zero

The AlphaGo Zero paper (free access) is more helpful:

Optimization: [...] Each mini­batch of data is sampled uniformly at random from all positions of the most recent 500,000 games of self­play.

For context, in total AlphaGoZero played 4.9 million games, so the buffer rotated about 10 times.

As another example, KataGo started with a small buffer early in training and then gradually grew it:

Samples are drawn uniformly from a growing moving window of the most recent data, with window size beginning at 250,000 samples and increasing to about 22 million by the end of the main run. See Appendix C for details.

(note that this is counting positions, not games like the AlphaZero paper)