0
$\begingroup$

I have written my own AlphaZero implementation and started training it recently.
Problem is, I am 99% sure there is a mistake and I do not know how to tackle this, since I cannot explain it. I am new too AI so my own go at debugging this wasn't quite succesful.

Input to my NN: A game state, represented by the board and position of the stones.
Output of my NN: a policy vector P and a scalar v(so an array and a number).
During self-play, training examples for each move are generated. These are later used to fit the network.

After having trained a bit, I can see both policy and value loss decreasing, which is good.
But for the very first game state (empty board) my prediction v of winning the game always stays at 0. This is very concerning, since the game I am training on is Connect4. Connect4 is a solved game and in the long run, the value for v should be 1(100% win chance).
So, any ideas what I can do? I mean I could post the code here, but that is quite a lot, I don't know if any of you are willing to read through it.

To show you what I mean, I'll show you the output of one of my test cases:

def test_p_v():
    game = connect4.Connect4()
    nnetwrapper = neuralnetwrapper.NNetWrapper(game, args)
    trainer = trainingonly1NN.Training(game = game, nnet= nnetwrapper, args = args)

    trainer.nnet.load_checkpoint(folder = './NNmodels/', filename='best.pth.tar')

    m = mctsnn.MCTS(nnet = nnetwrapper, args=args)

    m.root.expand()

    for c in m.root.children:
        print("P: {}  v: {}".format(c.state.P, c.state.v))

    print("Root MCTS: P: {}   v: {}".format(m.root.state.P, m.root.state.v))

results in:

P: [0.1436838  0.13809082 0.14174062 0.18597336 0.11126296 0.120884
 0.15836443]  v: [-0.14345692]
P: [0.14202288 0.13772981 0.14302546 0.1945151  0.11690026 0.1178078
 0.1479987 ]  v: [0.4222183]
P: [0.1447647  0.13066562 0.14334281 0.18055147 0.13374692 0.12126701
 0.1456615 ]  v: [-0.5827425]
P: [0.15192215 0.14221476 0.1443521  0.16634388 0.12634312 0.12711576
 0.14170831]  v: [-0.0229549]
P: [0.1456457  0.136381   0.13940862 0.17145196 0.12714048 0.12233274
 0.15763956]  v: [-0.02743456]
P: [0.15353182 0.13510287 0.1433772  0.16371183 0.12161442 0.1228981
 0.15976372]  v: [0.37902302]
P: [0.14321715 0.13596673 0.13836266 0.18927328 0.11999774 0.12481775
 0.1483647 ]  v: [-0.521353]
Root MCTS: P: [0.14296353 0.13863131 0.1358864  0.18102945 0.10981551 0.12779148
 0.16388236]   v: [0.]

So as you can see, for every different state, there are different P-values and different v-values, which makes sense.
It also makes sense for my ROOT Node to have the highest value in P at position 3, since this refers to the middle column.
But the v in my root Node is 0. This is alarming and I have no idea what do from here on.

I also checked some of the training examples passed to my neural network to learn, they look like this (board, P, Actual game result):

[[array([[0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0],
       [0, 0, 0, 0, 0, 0, 0]]), [0.13, 0.13, 0.15, 0.13, 0.18, 0.13, 0.15], 1]

whereas the very last number (1) is the v-value my network is supposed to fit! So it often even is 1!

But I fear that since this is the very root of all, the game result for the following notes is -1 and 1 changing every "step", so the average probably is 0 which is quite logical. Don't know how to express this, I fear it is trying to average the v mean of all states, instead of just training the v for one state.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.