# How are continuous actions sampled (or generated) from the policy network in PPO?

I am trying to understand and reproduce the Proximal Policy Optimization (PPO) algorithm in detail. One thing that I find missing in the paper introducing the algorithm is how exactly actions $$a_t$$ are generated given the policy network $$\pi_\theta(a_t|s_t)$$.

From the source code, I saw that discrete actions get sampled from some probability distribution (which I assume to be discrete in this case) parameterized by the output probabilities generated by $$\pi_\theta$$ given the state $$s_t$$.

However, what I don't understand is how continuous actions are sampled/generated from the policy network. Are they also sampled from a (probably continuous) distribution? In that case, which type of distribution is used and which parameters are predicted by the policy network to parameterize said distribution?

Also, is there any official literature that I could cite which introduces the method by which PPO generates its action outputs?

• If you have a new question, you should ask it in a different post, even though it's related to the current question (I actually don't know), because that may invalidate the existing answers.
– nbro
Commented Dec 16, 2020 at 19:35
• I think these answers I am searching for belong fundamentally together since only knowing what to predict does not make sense in absence of the knowledge about how to get to that prediction eventually. And for getting there, the surrogate loss needs to be considered as well since otherwise you don't have any way to properly train the model (in spite of knowing what it shall predict). And just for the context: $r_t(\theta)$ is an important part of the surrogate loss. But anyway. I sort of see your point, so let's make it a separate question then. Commented Dec 16, 2020 at 20:44
• Never mind. I think this question contains actually the answer to my edited question. I only hope that it is correct because this suspected answer is phrased as part of a question. But it sounds like a reasonable approach. So sorry for the confusion & I will 'revert' the edit. Commented Dec 16, 2020 at 20:54
• If you're not satisfied with that answer, eventually, you could ask a similar question but make sure to provide the context and say why you're not satisfied with that answer.
– nbro
Commented Dec 16, 2020 at 21:16

A common example for continuous spaces is the reparameterization trick, where your policy outputs $$\mu, \sigma = \pi(s)$$ and the action is $$a \sim \mathcal{N}(\mu, \sigma)$$.
• In the continuous case, how would the probability $\pi_\theta(a_t|s_t)$ be computed since we don't predict probability vectors any longer, but unconstrained real numbers instead? Just asking because we still need this to be able to compute the probability ratio $r(\theta)$. Commented Dec 14, 2020 at 12:06