How can I constraint the actions with dependent coordinates?

Question

I am working on a customized RL environment where each action is represented as a tuple $a = (a_1,a_2,\cdots,a_n)$ such that certain condition must be satisfied for entries of $a$ (for instance, $a_1+a_2+\cdots+a_n \leq \text{constant}$).

I am using the policy gradient method, but I am having some difficulty modeling the underlying probability distribution of actions. Is there any work done in this direction?

For the constraint $a_1+a_2+\cdots+a_n \leq \text{constant}$, I was thinking about generating $n+1$ uniform random variables $U_1,U_2,\cdots,U_n, U$, and set $a_i = \text{constant}\times U \times \frac{U_i}{\sum_{j=1}^n U_j}$. Problem is that the joint density is a bit messy to calculate, which is needed to get the negative log likelihood. I am curious about how such issue is handled in practice.

Answer 1

At first glance, I thought this was similar to "continuous-discrete" action selection (https://arxiv.org/pdf/1810.06394.pdf). However, I think your problem is different.

I am assuming that each $a_i$ is continuous and that the action which interacts with your environment is the entire vector $a = (a_1,a_2,\dotso,a_n)$ and not an individual $a_i$. Then you could treat it like a hierarchical problem. If you want $a_1 + a_2 < 2$ for example, then you could sample $a_1 \sim U(0,2)$ and $a_2 | a_1 \sim U(0, 2-a_1)$ and have $p(a) = p(a_2 | a_1)p(a_1)$. The specifics of how you do this depends more finely on how your problem is set up.

Perhaps you can find similar ideas from the paper linked above. Also, other work in the robitics literature studies structured and hybrid action spaces.