# How can I constraint the actions with dependent coordinates?

I am working on a customized RL environment where each action is represented as a tuple $$a = (a_1,a_2,\cdots,a_n)$$ such that certain condition must be satisfied for entries of $$a$$ (for instance, $$a_1+a_2+\cdots+a_n \leq \text{constant}$$).

I am using the policy gradient method, but I am having some difficulty modeling the underlying probability distribution of actions. Is there any work done in this direction?

For the constraint $$a_1+a_2+\cdots+a_n \leq \text{constant}$$, I was thinking about generating $$n+1$$ uniform random variables $$U_1,U_2,\cdots,U_n, U$$, and set $$a_i = \text{constant}\times U \times \frac{U_i}{\sum_{j=1}^n U_j}$$. Problem is that the joint density is a bit messy to calculate, which is needed to get the negative log likelihood. I am curious about how such issue is handled in practice.

I am assuming that each $$a_i$$ is continuous and that the action which interacts with your environment is the entire vector $$a = (a_1,a_2,\dotso,a_n)$$ and not an individual $$a_i$$. Then you could treat it like a hierarchical problem. If you want $$a_1 + a_2 < 2$$ for example, then you could sample $$a_1 \sim U(0,2)$$ and $$a_2 | a_1 \sim U(0, 2-a_1)$$ and have $$p(a) = p(a_2 | a_1)p(a_1)$$. The specifics of how you do this depends more finely on how your problem is set up.
• Thanks for the answer! Only issue for your method here is $a_1$ and $a_2$ are asymmetrical (e.g. $a_1$ always has higher mean that $a_2$). I kinda would like to treat each entry $a_i$ in a equal manner. I would definitely take a look a the paper you provided. I think policy gradient method can handle hybrid action space as long as the underlying probability distribution is properly specified. – Ryan Feb 27 at 14:08
• This is very contingent on the structure of your problem. Why not naively set $a_i \sim U(0,\frac{1}{n})$? I can't think of another general and non-hierarchical approach to this. If you do not want a systematic bias, then perhaps you could randomly sample the order for the indicies as well. – Alex L Feb 28 at 2:10
• Having $a_i$ being uniform(0,1\n) will impose a 1/n bound on them, in addition, they will be independent. Remember I would like to have dependent entries... I only have one feasible way of doing so which I mention in my post. – Ryan Feb 28 at 14:09