Why does Alpha Zero's Neural Network flip the board to be oriented towards the current player?

Question

While reading the AlphaZero paper in preparation to code my own RL algorithm to play Chess decently well, I saw that the

"The board is oriented to the perspective of the current player."

I was wondering why this is the case if there are two agents (black and white). Is it because there is only one central DCNN network used for board and move evaluation (i.e. there aren't two separate networks/policies used for the respective players - black and white) in the algorithm AlphaZero uses to generate moves?

If I were to implement a black move policy and a white move policy for the respective agents in my environment, would reflecting the board to match the perspective of the current player be necessary since theoretically the black agent should learn black's perspective of moves while the white agent should learn white's perspective of moves?

Answer 1

There is a single neural network that guides self-plays in the Monte Carlo Tree Search algorithm. The neural network gets the current state of the board $s$ as an input and outputs current policy $\pi(a|s)$ and value $v(s)$.

The action probabilities are encoded in a (8,8,73) tensor. First two dimensions encode the coordinates of the figure to "pick" from the board. The third dimension encode where to move this figure: check out this question for a discussion on how all possible moves are encoded in a 73 dimensional vector.

Similarly, the inputs of the network are organized in the (8, 8, 14 * 8 + 7 = 119) tensor. The first two 8 x 8 dimensions, again, encode the positions on the board. Then the positions of the figures one plane per 6 figure types: first 6 planes for player's figures, next 6 planes for opponent's figures and two repetition planes. The 14 planes are repeated 8 times supplying predecessor positions to the network. Finally, there are 7 extra planes encoding as a single uniform value over the board - castling rights (4 planes), total move count (2 planes) and the current player color (1 plane).

Note that the positions of player's figures and opponent's figures are encoded in fixed layers of the state tensor. If you don't flip the board to the perspective of the player then the network will have very different training inputs for black and white states. It also will have to figure out which direction the pawns can move depending on the current player color. None of that it is impossible, of course - but that unnecessarily complicates something that is already a very hard problem for the DNN to learn.

You can go further and completely split the training for white and black players, as you've described. But that'll essentially double the work you'll have to do train your nets (and, I suspect, there would be some stability troubles typical for adversarial training).

To summarize - you are generally right - there is no fundamental need to flip the board. All the above details in state encoding are done to simplify the learning task for the deep neural network.

Answer 2

I am not an expert in RL. I have been playing Go for some years.

Let's quote from AlphaZero's paper first:

Aside from komi, the rules of Go are also invariant to colour transposition; this knowledge is exploited by representing the board from the perspective of the current player (see Neural network architecture).

In the Game of Go, the difference between Black and White except the board representation is the komi (the amount of points that Black has to compensate White in the final count for playing first). Except the presence of komi, there should be no difference in strategy under the same position if colours exchanged. In other words, given a state $s$ of black stones and white stones on the board, if the optimal policy of Black playing first is $\pi$, then if colours of stones on the board exchanged and it is White's turn, the optimal policy for White should be the same as $\pi$.

With this in consideration, there are at least 2 advantages of using a network that represents the board in the perspective of Self/Opponent rather than Black/White.

The first is that it prevents the network from the possibility of giving inconsistent strategies under two representations of the same state. Consider a network $f_\theta$ that accepts the board representation in the order of $(B,W)$, and a state $s = (X_t,Y_t)$ in which $X_t$ is a feature map for black stones and $Y_t$ is a feature map for white stones and it is black's turn. Now consider a state $s' = (Y_t,X_t)$ (i.e. colours flipped) and it is white's turn. $s$ and $s'$ are essentially representation of the same state (except Komi which does not affect optimal policy). There could be a possibility that the network $f_\theta$ gives different policies for these two representations. However, if $f_\theta$ accepts the state as $(Self,Opponent)$, the input to the network would be the same (except the komi feature).

Therefore, this representation would significantly reduce number of states represented by the features vector $(X_t,Y_t)$, which would be the second advantage to training the neural network. If we consider that in Go, the same local position could appear in exchanged colour in another position, the network could by this implementation, recognize them as the same position. A decrease in the number of states could mean a significant drop in parameters and power needed from the network.

The same principle of making use of different representations of the same state is followed in AlphaGo's other training implementations as well, such as augmenting its training data to include rotations and reflections of the same board position.

However, in the game of Chess, this would be a different case. For a chess position, if the pieces' colours are exchanged and it becomes opponent's turn, it would be a different state because the positions of the KING and the QUEEN are not the same for the two colours.

Answer 3

In board games, whose turn is it play makes a significant difference in the outcome of the game. Many times its a difference of loss or a win.

So this information has to be communicated to the neural network. There are two ways to communicate that information:

1. To add extra input nodes which say whose turn it is to make move.

2. To simply flip the signs of the pieces.

Method 1 works but it increases the number of nodes hence making the neural network more complex and larger. So method 2 is preferred to keep the neural network simple without losing the needed information. This process is known as canonical form of the board.

Flipping signs wouldn't work for games that have more than two players.