How can I use one neural network for both players in Alpha Zero (Connect 4)?

Question

First of all, it is great to have found this community!

I am currently implementing my own Alpha Zero clone on Connect4. However, I have a mental barrier I cannot overcome.

How can I use one neural network for both players? I do not understand what the input should be.

Do I just put in the board position ($6 \times 7$) and let's say Player1's pieces on the board are represented as $-1$, empty board as $0$ and Player2's pieces as $1$?

To me, that seems the most efficient. But then, in the backpropagation, I feel like this cannot be working. If I update the same network for both players (which Alpha Zero does), don't I try to optimize Player1 and Player 2 at the same time?

I just can't get my head around it. 2 Neural networks, each for one player is understandable for me. But one network? I don't understand how to backpropagate? Should I just flip my "z" (the result of the game) every time I go one layer backward? Is that all there is to using one network?

I hope I made this clear enough. I am quite confused, I tried my best.

Thank you for reading this!

Answer 1

Let's define your problem from another point of view. Let's say that in this RL problem you have two agents (agent1 and agent2) that compete with each other in order to accomplish their own goal, i.e., wining connect4 game.

Therefore, we could say that from agent1's point of view, he is player1 and the player2 is agent2. The same way, from agent2's point of view, he is player1 and the player2 is agent1.

Here, we can use the same definition as you proposed:

"let's say player1's pieces on the board are represented as -1, empty board as 0 and player2's pieces as 1"

When it is the turn for agent1, he will receive an observation (representing the board configuration) from the environment. In this case, agent1's pieces are represented as -1, empty board as 0 and for player2's (agent2's) pieces as 1. Then, you could use the neural network to compute, e.g., the next action for agent1.

When it is the turn for agent2, he will receive another observation from the environment, but this time you represent its own pieces as -1, while the agent1's (player2's from agent2's point of view) as 1.

With this approach, the neural network will always receives as an input a representation in which the pieces 'it' controls are -1, while its rival's pieces are 1 and empty space 0. From the neural network's point of view, he does not care about which agent is playing as the actual observation is indistinguishable to calculate the next action.

Therefore, you will have to generate two different observations, in which you invert the sign of the ones on it, depending on which turn it is (agent1's or agent2's turn).

For example: