Regarding the AlphaZero paper, it is not clear to me when the Monte Carlo Tree Search (MCTS) results will be cleaned up.
I assume this has to happen at some point, since mixing results could lead to lower quality results? Imagine in the self-play the Neural Network (NN) is updated to a new version and evaluates certain patterns differently by detecting a new trick. Many iterations must follow to outperform the old best choice (visit-count). I imagine discarding old MCTS results should be done about between an episode and the next NN weight updates.
I feel that a wrong decision here could have a strong negative impact on the overall learning process.