I am working on a customized RL environment where each action is represented as a tuple $a = (a_1,a_2,\cdots,a_n)$ such that certain condition must be satisfied for entries of $a$ (for instance, $a_1+a_2+\cdots+a_n \leq \text{constant}$).

I am using the policy gradient method, but I am having some difficulty modeling the underlying probability distribution of actions. Is there any work done in this direction?

For the constraint $a_1+a_2+\cdots+a_n \leq \text{constant}$, I was thinking about generating $n+1$ uniform random variables $U_1,U_2,\cdots,U_n, U$, and set $a_i = \text{constant}\times U \times \frac{U_i}{\sum_{j=1}^n U_j}$. Problem is that the joint density is a bit messy to calculate, which is needed to get the negative log likelihood. I am curious about how such issue is handled in practice.


At first glance, I thought this was similar to "continuous-discrete" action selection (https://arxiv.org/pdf/1810.06394.pdf). However, I think your problem is different.

I am assuming that each $a_i$ is continuous and that the action which interacts with your environment is the entire vector $a = (a_1,a_2,\dotso,a_n)$ and not an individual $a_i$. Then you could treat it like a hierarchical problem. If you want $a_1 + a_2 < 2$ for example, then you could sample $a_1 \sim U(0,2)$ and $a_2 | a_1 \sim U(0, 2-a_1)$ and have $p(a) = p(a_2 | a_1)p(a_1)$. The specifics of how you do this depends more finely on how your problem is set up.

Perhaps you can find similar ideas from the paper linked above. Also, other work in the robitics literature studies structured and hybrid action spaces.

  • $\begingroup$ Thanks for the answer! Only issue for your method here is $a_1$ and $a_2$ are asymmetrical (e.g. $a_1$ always has higher mean that $a_2$). I kinda would like to treat each entry $a_i$ in a equal manner. I would definitely take a look a the paper you provided. I think policy gradient method can handle hybrid action space as long as the underlying probability distribution is properly specified. $\endgroup$ – Ryan Feb 27 '20 at 14:08
  • $\begingroup$ This is very contingent on the structure of your problem. Why not naively set $a_i \sim U(0,\frac{1}{n})$? I can't think of another general and non-hierarchical approach to this. If you do not want a systematic bias, then perhaps you could randomly sample the order for the indicies as well. $\endgroup$ – Alex L Feb 28 '20 at 2:10
  • $\begingroup$ Having $a_i$ being uniform(0,1\n) will impose a 1/n bound on them, in addition, they will be independent. Remember I would like to have dependent entries... I only have one feasible way of doing so which I mention in my post. $\endgroup$ – Ryan Feb 28 '20 at 14:09
  • $\begingroup$ Right, I should have added general, non-hierarchical and tractable. Since you are using policy gradient, don't you need to parameterize the distribution anyway? Perhaps you can simplify the problem by assuming IID while using lagrange multipliers to constrain the parameters. $\endgroup$ – Alex L Feb 29 '20 at 1:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.