3
$\begingroup$

I am experimenting with a self-programmed version of AZ Chess following the described methodology in the official paper. I am experiencing that at the beginning of the self-play (when the ANN weights are is still randomly initialized) a very high portion of all games end with a "draw" (ca. 97% of all games). This is not surprising because if two pure random players play against each other, it is very unlikely that white or black achieve a win/lose. As a result, ca. 97% of all board states visited during the sell-play are trained towards a target value of 0 (instead towards +1/-1 for white to win/lose). This means that there is very litte training data (3% of all training data generated via self-play) that helps the ANN to learn how to win and to get stronger.

Does anyone know how DeepMind has overcome this problem? I have found nothing about this in the www.

Improve this question
$\endgroup$
5
  • $\begingroup$ Hello and welcome to AI Stack Exchange! How many games have you currently run during training? $\endgroup$
    – DeepQZero
    Commented Jan 23 at 16:22
  • $\begingroup$ Also, are you using a MCTS? $\endgroup$
    – DeepQZero
    Commented Jan 23 at 16:41
  • $\begingroup$ I have already run ca. 1000 self-play games. But that is enough to see that the vast majority end with a "draw" which does not generate any training effect. $\endgroup$
    – Oliver Ewald
    Commented Jan 23 at 21:04
  • $\begingroup$ Yes, I general I use MCTS during self-play, but at the beginning I set "number of roll-outs" to just 1 in order to speed up the process (that led to the 97% draws). But during the last 24 hours I set "number of roll-outs" to 800 and observed a decrease of "draws" (down from 97% to 80%). But the 80% still seem far suboptimal for effective learning. I am wondering if it was really the huge number of self-play games that solved the problem in the original version of AlphaZero, or do I still miss something conceptionally-wise? $\endgroup$
    – Oliver Ewald
    Commented Jan 23 at 21:13
  • $\begingroup$ @OliverEwald AlphaZero training with only a single visit is a bit strange, how would the policy head ever learning anything? You definitely need at least some rollouts! $\endgroup$
    – KarelPeeters
    Commented Jan 25 at 22:24

1 Answer 1

Reset to default
1
$\begingroup$

First intuition

I wrote a quick script to test the outcome distribution of random chess games: I get about 7% wins for black and white each, so 14% non-drawn games. The MCTS should still be able to find simple mate in one moves even with a fully uninitialized network, so I would expect it to draw even less.

Your 97% draw rate is significantly higher than that, so I suspect something else is going wrong. Some ideas to try:

  • Do your own random moves test to confirm that you also get 7% win for each player, otherwise something went wrong with your board implementation.
  • Run MCTS with a random NN on a mate-in-one position, and ensure that it finds the mate and picks it as the best (highest visit count) move after a couple thousand visits.

I think in theory 3% decisive games is already enough to kickstart the learning process, but specifically for chess it probably means something in the implementation is wrong.

Actual data

Here's the win/draw/loss plot from a chess run of my own AlphaZero implementation, kZero. The horizontal axis corresponds to about 300k selfplay games played. This is with 400 visits/move, and settings otherwise mostly similar to the AlphaZero paper.

It looks like the behavior is as follows:

  • The initial games generated with a random NN have only 17% draws, so MCTS is finding a lot of mates
  • Very quickly the drawrate goes up to 35%, as the NN learns to avoid mate in one
  • After that the drawrate drops back down to 8%, as the NN learns to finish games from winning positions.
  • For the rest of the run the drawrate slowly climbs again, as is typical for stronger chess players and engines.

WDL plot

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks a lot for your quick and comprehensive answer! $\endgroup$
    – Oliver Ewald
    Commented Jan 24 at 19:41
  • $\begingroup$ 1. I will do new random checks (without MCTS) to see if my win ratios will go up towards your 14% 2. I just checked that my MCTS clearly detects "mate in one" 3. What I find surprising is that your MCTS on a fully uninitialized NN only finds 17% draws. That's far off from my 97% 4. Currently I define a game as "drawn" if there is no mate after 100 moves. What rules do you apply to avoid endless games? 5. I will answer back here once I have 1. completed with (possibly) adjusted rules how to define draws. $\endgroup$
    – Oliver Ewald
    Commented Jan 24 at 19:57
  • 1
    $\begingroup$ @OliverEwald I only terminate games based on 3-fold repetition, the 50 move rule and insufficient material (only king vs king). For this run the average game length started at 100, peaked at 300 and then dropped back down to 110. Cutting of games at 100 moves indeed causes lots of draws, so that's probably your main issue! $\endgroup$
    – KarelPeeters
    Commented Jan 25 at 22:23
  • $\begingroup$ Thanks a lot for clarification! I will test with modified termination rules. $\endgroup$
    – Oliver Ewald
    Commented Jan 26 at 8:09

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .