# PPO: multiple discrete actions per step, one depends on the other

I have a custom PPO implementation, and it works fine, but I need to add to it the ability to select 2 actions per turn, one different in nature from the other, one dependent on the other.

Imagine that a turn a had 20 possible actions. Now, for each of these action of type A, I need to choose one from 3 possible actions B. They are not the same, and thus they can not be on different turns.

What I tried was to flatten the 20x3 space to a 60. Then, action A is 60//3 and action B is 60%3. But this does not train well. Are there any good methods for this issue?

• Hi @Antonis Karvelas! Could you explain what you mean by the algorithm not training well? Thanks! Jun 9, 2022 at 20:03
• Nothing converges. I imagine that it can pick good action A and bad action B or vice versa but it can't figure out which is which... Jun 9, 2022 at 20:05

In the two step process you describe, you would have something like \begin{align*} \mathbb{P}(a_t) = \mathbb{P}(a_{(t, 1)}=A)\mathbb{P}(a_{(t, 2)}=B \, | \, a_{(t, 1)}=A) \end{align*} where $$a_t$$ is the action taken, which consists of the two steps $$a_{(t,1)}$$ and $$a_{(t,2)}$$.