Q learning: How to create output layer in which actions are combinations of multiple sub-actions

Question

Suppose in my example I want an agent to learn a behavior that is made up of a combination of actions.

So consider the following example with a tamagotchi like game: There are 5 pets and 3 actions that can be taken for each pet (give food, give water, play).

Now I know that in the standard approach, the DQN network would be such that the output is of shape 5x3 = 15. But for a larger space of pets and possible actions this would become unfeasible. So is there an adaption of deep q learning in which I could use a network that has only 3+5=8 output layers and treats the selection of the pet and the action independently?

Answer 1

What you're describing is known as a composite action space (or a factorisable action space). The key assumption is that the action space can be factorised in sub-action spaces, i.e. $\mathcal{A} = \mathcal{A}_1 \times ... \times \mathcal{A}_N$.

As far as I know, this is a relatively new area in terms of research contributions. The two papers that come to mind are action branching networks and Decoupled Q-Network.

The latter is likely to be the easiest to implement. Rather than flattening your sub-action spaces into an action space that consists of very many primitive actions and having a DQN provide an output value for the many different possibilities, instead you have $N$ output heads for each sub-action space. Where a DQN would be a function $f : \mathcal{S} \rightarrow \mathbb{R}^{|\mathcal{A}|}$, the decoupled network would be $N$ functions $f_i : \mathcal{S} \rightarrow \mathbb{R}^{|\mathcal{A}_i|}$. If we refer to the these sub-functions as utility functions, i.e. $U_i(s, a_i)$, then we can obtain the Q-value as $Q(s, a_1, ..., a_N) = \sum_{i=1}^N U_i(s, a_i)$. This way, the size of the spaces we need to learn values for grows linearly with the number of sub-action spaces rather than exponentially.