1
$\begingroup$

Specifically for continuous control PPO, let's say my action space range is between $X$ (low) and $Y$ (high) and they are all sampled from a Gaussian Action Distribution with mean $\mu$ and standard deviation $\rho$.

From what I understood, the actions sampled should fall between $\mu - \rho$ and $\mu + \rho$, but that's not what happens in practice? What am I misunderstanding here? How do I ensure this range constraint from a custom action distribution with a given mean and standard deviation?

Any advice or tips for me? I would really appreciate any insights!

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

First of all, the support of a normal distribution is the entire real line (or, in general, $\mathbb{R}^n$ for an $n$-dimensional multivariate normal distribution) so your action can be any number in $\mathbb{R}$. What you may be getting confused with is that with probability 0.68 you will obtain an action that is within +/- 1 standard deviation from the mean.

Now, to answer the question of how you can do this using RL:

To use the Normal distribution in this setting I would simply clip my actions in the environment. If, for example, the actor gives an action below your minimum value $X$, lets say it gives $X-0.5$, then I would simply clip the action to be $X$ when executed in the environment. This way your actor can still sample from a normal distribution which could give answers below $X$ (or above $Y$) and be used with your environment.

If, for instance, your desired range was $(-1, 1)$ then another option would be to define your distribution to be $Y = \mbox{tanh}(X)$, where $X \sim N(\mu, \sigma)$. You can then find the density function of $Y$ using e.g. the density transformation method.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I see, thank you so much for clearing that about the normal distribution. From what I understand, the normal distribution is just a special case of the Gaussian distribution? Is that correct? What if, for example, the output of the actor leads to an action range that is much smaller than the actual range of actions. That is, what if the range of output is [-2, 2] but the actual action range is [-5, 5]. Is expanding the output (unsquashing it) into the original action range of [-5,5] a valid thing to do? $\endgroup$
    – hridayns
    Commented Jul 9, 2021 at 9:08
  • $\begingroup$ the normal distribution is the Gaussian distribution, it is just a different name that people use :) in that instance, I would recommend to do some kind of normalisation so that the actors range equals the actual environments range. If the actual environment range is symmetric, i.e. $[-x, x]$ then you could use a tanh activation function for the final layer of the network and multiply the output by $x$. $\endgroup$
    – David
    Commented Jul 9, 2021 at 10:16
  • $\begingroup$ Yeah, sadly my action range isn't symmetric! But thanks for the help! $\endgroup$
    – hridayns
    Commented Jul 9, 2021 at 10:28
  • $\begingroup$ you could try shifting you action range to start from 0, i.e. if the range is $[x, y]$ then make the range $[0, y-x]$ and then you just need to use a relu activation and clip at $y-x$, or use a sigmoid and multiply by $y-x$. $\endgroup$
    – David
    Commented Jul 9, 2021 at 10:48

You must log in to answer this question.