4
$\begingroup$

First of all, it is great to have found this community!

I am currently implementing my own Alpha Zero clone on Connect4. However, I have a mental barrier I cannot overcome.

How can I use one neural network for both players? I do not understand what the input should be.

Do I just put in the board position ($6 \times 7$) and let's say Player1's pieces on the board are represented as $-1$, empty board as $0$ and Player2's pieces as $1$?

To me, that seems the most efficient. But then, in the backpropagation, I feel like this cannot be working. If I update the same network for both players (which Alpha Zero does), don't I try to optimize Player1 and Player 2 at the same time?

I just can't get my head around it. 2 Neural networks, each for one player is understandable for me. But one network? I don't understand how to backpropagate? Should I just flip my "z" (the result of the game) every time I go one layer backward? Is that all there is to using one network?

I hope I made this clear enough. I am quite confused, I tried my best.

Thank you for reading this!

$\endgroup$
  • 2
    $\begingroup$ Hi and welcome to this community! We are glad to help whenever we can :) I've slightly edited the title of your question (for better readability). I hope it is not a problem. $\endgroup$ – nbro Jul 24 at 22:24
3
$\begingroup$

Let's define your problem from another point of view. Let's say that in this RL problem you have two agents (agent1 and agent2) that compete with each other in order to accomplish their own goal, i.e., wining connect4 game.

Therefore, we could say that from agent1's point of view, he is player1 and the player2 is agent2. The same way, from agent2's point of view, he is player1 and the player2 is agent1.

Here, we can use the same definition as you proposed:

"let's say player1's pieces on the board are represented as -1, empty board as 0 and player2's pieces as 1"

When it is the turn for agent1, he will receive an observation (representing the board configuration) from the environment. In this case, agent1's pieces are represented as -1, empty board as 0 and for player2's (agent2's) pieces as 1. Then, you could use the neural network to compute, e.g., the next action for agent1.

When it is the turn for agent2, he will receive another observation from the environment, but this time you represent its own pieces as -1, while the agent1's (player2's from agent2's point of view) as 1.

With this approach, the neural network will always receives as an input a representation in which the pieces 'it' controls are -1, while its rival's pieces are 1 and empty space 0. From the neural network's point of view, he does not care about which agent is playing as the actual observation is indistinguishable to calculate the next action.

Therefore, you will have to generate two different observations, in which you invert the sign of the ones on it, depending on which turn it is (agent1's or agent2's turn).

For example:

agent1's board

agent2's board

$\endgroup$
  • $\begingroup$ oh my, that makes a lot of sense. Is that what a canonical form is? So to be sure I got this right. My first layer will be exactly 7x6 = 42 Neurons, one for each position. Every time a player took a turn, the board for the next player is simply multiplied by -1, to flip the board. And at the end of the game, the result z € [-1,1] will be backpropgated as the same all the way, without flipping it, right? $\endgroup$ – Leviathan Jul 24 at 20:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.