In the AlphaZero learning algorithm, during self-play to generate training games, the move played is chosen with probability proportional to the MCTS visits raised to the $\tau$-th power, where $\tau$ is the so called temperature. Higher temperatures correspond to more exploration. It seems that in deepmind's original paper (on AlphaGo Zero if I'm not mistaken) it is mentioned that temperature is decayed to zero after move 30 in Go/Baduk, then this is contradicted in the AlphaZero with it saying that temperature is not decayed at all, and finally in AlphaZero's pseudocode I believe it is implied that the temperature is decayed after some number of moves. Specifically I believe that lczero concluded that they decayed after 15 moves for chess. It's not clear to me after searching what the current training regime for lczero is with regards to temperature. Also, I believe that ELF openGo efforts used $\tau=1$ for the entire game.

Question: Is there a consensus on what $\tau$ should be? Does it matter if the training is in early phases or not (i.e. if the AI is not advanced yet is it beneficial to explore seemingly "worse" moves?) How dependent on the game is this optimal $\tau$? If I have a game which lasts 50 moves average, and I want to decay $\tau$, is there a best practice?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.