0
$\begingroup$

Regarding the AlphaZero paper, it is not clear to me when the Monte Carlo Tree Search (MCTS) results will be cleaned up.

I assume this has to happen at some point, since mixing results could lead to lower quality results? Imagine in the self-play the Neural Network (NN) is updated to a new version and evaluates certain patterns differently by detecting a new trick. Many iterations must follow to outperform the old best choice (visit-count). I imagine discarding old MCTS results should be done about between an episode and the next NN weight updates.

I feel that a wrong decision here could have a strong negative impact on the overall learning process.

$\endgroup$

1 Answer 1

1
$\begingroup$

You're right, it would not be great to keep training on games played early during the selfplay process, since they would just hurt the network playing strength.

The AlphaZero paper itself does not elaborate, but it has this sneaky sentence:

Unless otherwise specified, the training and search algorithm and parameters are identical to AlphaGo Zero

The AlphaGo Zero paper (free access version) is more helpful:

Optimization: [...] Each mini­batch of data is sampled uniformly at random from all positions of the most recent 500,000 games of self­play.

For context, in total AlphaGoZero played 4.9 million games, so the buffer rotated around about 10 times.

As another example, KataGo, started with a small buffer early in training and then gradually grew it.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .