How to define a continuous action distribution with a specific range for Reinforcement Learning?

Specifically for continuous control PPO, let's say my action space range is between $$X$$ (low) and $$Y$$ (high) and they are all sampled from a Gaussian Action Distribution with mean $$\mu$$ and standard deviation $$\rho$$.

From what I understood, the actions sampled should fall between $$\mu - \rho$$ and $$\mu + \rho$$, but that's not what happens in practice? What am I misunderstanding here? How do I ensure this range constraint from a custom action distribution with a given mean and standard deviation?

Any advice or tips for me? I would really appreciate any insights!

First of all, the support of a normal distribution is the entire real line (or, in general, $$\mathbb{R}^n$$ for an $$n$$-dimensional multivariate normal distribution) so your action can be any number in $$\mathbb{R}$$. What you may be getting confused with is that with probability 0.68 you will obtain an action that is within +/- 1 standard deviation from the mean.
To use the Normal distribution in this setting I would simply clip my actions in the environment. If, for example, the actor gives an action below your minimum value $$X$$, lets say it gives $$X-0.5$$, then I would simply clip the action to be $$X$$ when executed in the environment. This way your actor can still sample from a normal distribution which could give answers below $$X$$ (or above $$Y$$) and be used with your environment.
If, for instance, your desired range was $$(-1, 1)$$ then another option would be to define your distribution to be $$Y = \mbox{tanh}(X)$$, where $$X \sim N(\mu, \sigma)$$. You can then find the density function of $$Y$$ using e.g. the density transformation method.
• the normal distribution is the Gaussian distribution, it is just a different name that people use :) in that instance, I would recommend to do some kind of normalisation so that the actors range equals the actual environments range. If the actual environment range is symmetric, i.e. $[-x, x]$ then you could use a tanh activation function for the final layer of the network and multiply the output by $x$. Jul 9 '21 at 10:16
• you could try shifting you action range to start from 0, i.e. if the range is $[x, y]$ then make the range $[0, y-x]$ and then you just need to use a relu activation and clip at $y-x$, or use a sigmoid and multiply by $y-x$. Jul 9 '21 at 10:48