Why does each component of the tuple that represents an action have a categorical distribution in the TRPO paper?

Question

I was going through the TRPO paper, and there was a line under Appendix D "Approximating Factored Policies with Neural Networks" in the last paragraph which I am unable to understand

The action consists of a tuple $(a_1, a_2..... , a_K)$ of integers $a_k\in\{1, 2,......,N_k\} $ and each of these components is assumed to have a categorical distribution.

I can't seem to get how each component has a categorical distribution. I think it should be the tuple that has a categorical distribution. I think I am getting something wrong.

Answer 1

I'm not sure specifically which Atari games present this type of action space, but you can imagine a game in which you can perform multiple different types of actions at the same timestep (i.e. the different "factors" they mention in the paper).

As an example, imagine a game in which you can both move and jump at the same time. In that case, you might have a 4-dimensional discrete action space for moving (NSWE), and a 2-dimensional discrete action space for jumping (yes/no jump), both of which will require a Categorical distribution which has the size of that factor ($N_k$ in the paper).

So in this case, you would need to have a categorical distribution for each factor, unless you were to turn these two factors into 1 joint 4*2-dimensional factor and learn a single categorical distribution on that (which would likely be less efficient).