MuZero seems to use two different methods to encode actions into planes for Atari games:

  1. For the input action to the representation function, MuZero encodes historical actions as simple bias planes, scaled as $a/18$, where $18$ is the total number of valid actions in Atari.(from the appendix E of the paper)
  2. For the input action to the dynamics function, Muzero encode an action as a one-hot vector, which is tiled appropriately into planes(from the appendix F of the paper)

I'm not so sure about how to make of the term "bias plane".

About the second, my understanding is that, as an example, for action $4$, we first apply one-hot encoding, which gives us a zero vector of length $18$ with one in the $5$-th position(as there are $18$ actions). Then we tile it and get a zero vector of length $36$, with ones in the $5$-th and $23$-rd positions. At last, this vector is reshaped into a $6\times 6$ plane as follows:

$$ 0, 0, 0, 0, 1, 0\\ 0, 0, 0, 0, 0, 0\\ 0, 0, 0, 0, 0, 0\\ 0, 0, 0, 0, 1, 0\\ 0, 0, 0, 0, 0, 0\\ 0, 0, 0, 0, 0, 0 $$

  1. The bias plane is a layer everywhere equal to the constant $a/18$ where $a$ is the action. So each of the 32 frames has three frames for RGB and a fourth frame which is the bias plane making for 128 input layers. This is explained in the Network Architecture section where it mentions that the actions are "broadcast" to the planes.

  2. For this I do not have conclusive evidence, but I think tiling a vector means arranging parallel copies of it into e.g. a grid shape. In other words, the input is 6x6x18, and action 1 is represented as all ones in the first plane with all zeros in the remaining planes. One problem with the way you've depicted is that the input is subject to convolutions, but there is no inherent reason why the fifth position and eleventh actions (which are next to each other vertically) should be included in the same 3x3 filter application, but the fifth and seventh (for example) actions should not.

  • $\begingroup$ Hi @expz. Thank you for the explanation. I think you are right, but I'm a little confused about why we use different representations for actions in representation and dynamics functions? $\endgroup$ – Maybe Jun 21 '20 at 1:14
  • $\begingroup$ If you were to use the one hot encoding for the representation network input then there would be 18 planes per frame or 576 planes devoted to actions. I assume they wanted to reduce that to something more manageable. However the fact that they use the one hot encoding instead of the more succinct encoding for the dynamics network which has a similar architecture suggests to me that perhaps the one hot encoding performs better when it is practical to use. $\endgroup$ – expz Jun 24 '20 at 3:12
  • $\begingroup$ That makes sense. Thank you so much! $\endgroup$ – Maybe Jun 25 '20 at 23:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.