I am trying to reproduce AlphaZero's algorithm on the board game Carcassonne. Since I want to use the final game score differences (i.e. victory point of player 1 - victory point of player 2) as the final and only reward, AlphaZero's UCB score can no longer work since it assumed that the game reward is either 1 or -1. Therefore, I am using the trick from MuZero, which, when computing UCB score, normalizes the child value estimate to the range $[0, 1]$ using the minimum and maximum value estimates in the entire tree.
However, after implementing this, my algorithm almost never get past the first iteration, since the win-rate against last iteration is almost never higher than 50% (I am using an arena threshold of 0.55). If I don't normalize the reward, then the win rate seems to be higher.
My hypothesis is that, in the first few MCTS searches, there are not many estimates in the tree, and a few of the node values will be normalized to 0 or 1, which are very extreme, despite their original model-predicted values being very small in magnitude. This may cause the tree to overfit to these values instead of exploring other nodes.
- Is this a common issue with self-play, or is it indicative of a bug in my implementation?
- What are some common ways to overcome the issue (e.g. increasing data, MCTS iterations, etc)? If the algorithm does not get past the first arena, then it will never progress since it always gets reset to the initial model.