# Is there any research on the application of policy gradients to problems where the selection of an action requires the selection of another one?

I am working on a problem and want to explore if it can be solved with PPO (or other policy gradient methods). The problem is that the action space is a bit special, compared to classic RL environments.

At each time $$t$$, we chose between 4 actions: $$a_1\in \{0, 1, 2, 3\}$$, but given $$a_1 = 0 \text{ or } 2$$, we need to chose three more actions: $$a_2, a_3, a_4$$ (which all three can be chosen from categorical distributions).

I know I can design this kind of policy myself and re-work the entropy terms, and so on for PPO.

My question is: is there any research into this kind of RL?

I am having a hard time finding someone working with problems in which the actions chosen are dependent on other chosen at the same time. I have looked into Hierarchical RL, but the papers I have found have not worked with this particular kind of problem.

If these action spaces were small ($$a_2, a_3$$ are chosen from categorical distributions with $$\sim$$800 different options), one solution would be to roll it out into one big policy where each possible combination of actions is represented by one choice in the policy. But my concern of doing this with a bigger action space is that the choice of $$a_1 = 1, 3$$ where we don't choose the other separate actions will get lost in the policy.

• you could check out the options framework (I think mainly introduced by Doima Precup in her PhD work) but it works somewhat similar to what you are describing; at time $t$ you choose an option which then dictates that you follow an options policy until some stopping criteria is met (its been a while since I looked at this so I may not be being 100% accurate, but the general idea should be right). Apr 6 at 12:57
• I made changes to your post to improve the clarity. Make sure the meaning has not changed.
– nbro
Apr 9 at 3:39