1
$\begingroup$

I have a question regarding the action space of the policy network used in AlphaZero.

From the paper:

We represent the policy π(a|s) by a 8 × 8 × 73 stack of planes encoding a probability distribution over 4,672 possible moves. Each of the 8 × 8 positions identifies the square from which to “pick up” a piece. The first 56 planes encode possible ‘queen moves’ for any piece: a number of squares [1..7] in which the piece will be moved, along one of eight relative compass directions {N,NE,E,SE,S,SW,W,NW}......The policy in Go is represented identically to AlphaGo Zero (29), using a flat distribution over 19 × 19 + 1 moves representing possible stone placements and the pass move. We also tried using a flat distribution over moves for chess and shogi; the final result was almost identical although training was slightly slower.

I don't understand why a stack of planes is used for the action space here. I'm also not entirely sure I understand how this representation is used. My guess is that for Chess, the 8x8 plane represents the board, and each square has a probability assigned to it of picking up a piece on that square (let's assume that all illegal moves haven't been masked yet, so all squares have probability mass on them). From there, we choose from possible 'Queen' type moves or 'Knight' type moves, which total to 73 different types of moves. Is this interpretation correct? How would one go from this representation to sampling a legal move (i.e. how is this used to parametrize a distribution I can actually sample moves from?)

During MCTS when expanding a leaf node, we get $p_a$, the probability of taking some action $a$ from the policy head, so I would also need to be able to go from this 'planes' representation to the probability of taking a specific action.

The paper also mentions trying out 'flat distributions', which I'm not entirely sure what this means either.

$\endgroup$
1
  • $\begingroup$ Can you please put your main specific question in the title? $\endgroup$ – nbro Dec 16 '20 at 17:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.