1
$\begingroup$

I'm trying to build neural networks for games like Go, Reversi, Othello, Checkers, or even tic-tac-toe, not by calculating a move, but by making them evaluate a position.

The input is any board situation. The output is a score or estimate for the probability to win, or how favorable a given position is. Where 1 = guaranteed to win and 0 = guaranteed to lose.

In any given turn I can then loop over all possible moves for the current player, evaluate the resulting game situations, and pick the one with the highest score.

Hopefully, by letting this neural network play a trillion games vs itself, it can develop a sensible scoring function resulting in strong play.

Question: How do I train such a network ?

In every game, I can keep evaluating and making moves back and forth until one of the AI players wins. In that case the last game situation (right before the winning move) for the winning player should have a target value of 1, and the opposite situation (for the losing player) has a target value of 0.

Note that I don't intend to make the evaluation network two-sided. I encode the game situation as always "my own" pieces vs "the opponent", and then evaluate a score from my own (i.e. the current player's) side or perspective. Then after I pick a move, I flip sides so to speak, so the opponent pieces now become my own and vice versa, and then evaluate scores again (now from the other side's perspective) for the next counter move.

So the input to such a network does explicitly encode black and white pieces, or naughts and crosses (in case of tic-tac-toe) but just my pieces vs their pieces. And then evaluates how favorable the given game situation is for me, always assuming it's my turn.

I can obviously assign a desired score or truth value for the last move in the game (1 for win, 0 for loss) but how do I backpropagate that towards earlier situations in a played game?

Should I somehow distribute the 1 or 0 result back a few steps, with a decaying adjustment factor or learning rate? In a game with 40 turns, it might make sense to consider the last few situations as good or bad (being close to winning or losing) but I guess that shouldn't reflect all the way back to the first few moves in the game.

Or am I completely mistaken with this approach and is this not how it's supposed to be done?

New contributor
RocketNuts is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$
  • $\begingroup$ This idea is on the right track! you have pretty much defined alphaZero, where they train a NN to evaluate each move, and then use MC-search for the actual play throughs! $\endgroup$ – mshlis Dec 2 at 14:57
  • $\begingroup$ Ok thanks, and do you know if I can find some explanation or further documentation on the training process? $\endgroup$ – RocketNuts Dec 2 at 20:45
0
$\begingroup$

The evaluation of the last steps in the game can be made with the 1 and 0 as you said. For all the other steps, the evaluation should be the evaluation of the best next step with a small decay.

$\endgroup$
  • $\begingroup$ Thanks, how do you mean the evaluation of the best next step? For example, if a simulated game of the neural network vs itself proceeds like this. The last move of A (in step 7) should obviously be scored 1 because it wins the game, and the preceding move of B (in step 6) should score 0 because it allowed the opponent to win. But then how further? The preceding step directly before that, move 5 by player A, was actually a very bad move, as player B could have won after that. And in fact player A started the game very badly. So how do I further assign scores? $\endgroup$ – RocketNuts 14 hours ago

Your Answer

RocketNuts is a new contributor. Be nice, and check out our Code of Conduct.

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.