I am trying to understand and reproduce the Proximal Policy Optimization (PPO) algorithm in detail. One thing that I find missing in the paper introducing the algorithm is how exactly actions $a_t$ are generated given the policy network $\pi_\theta(a_t|s_t)$.

From the source code, I saw that discrete actions get sampled from some probability distribution (which I assume to be discrete in this case) parameterized by the output probabilities generated by $\pi_\theta$ given the state $s_t$.

However, what I don't understand is how continuous actions are sampled/generated from the policy network. Are they also sampled from a (probably continuous) distribution? In that case, which type of distribution is used and which parameters are predicted by the policy network to parameterize said distribution?

Also, is there any official literature that I could cite which introduces the method by which PPO generates its action outputs?

Edit:

A question which I think is immediately related to the original question is how the probability ratio $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)}$ (as stated in the aforementioned paper) is calculated when working with continuous outputs generated by the policy network $\pi_\theta$.

Thus, given that $\pi_\theta$ predicts some mean $\mu$ and standard deviation $\sigma$ as outputs rather than a vector of discrete probability values, how is $r_t(\theta)$ calculated (since there are no probability value estimates $\pi_\theta(a_t|s_t)$ being generated any longer while $r_\theta$ is clearly called a probability ratio in the discrete case)?

I am trying to understand and reproduce the Proximal Policy Optimization (PPO) algorithm in detail. One thing that I find missing in the paper introducing the algorithm is how exactly actions $a_t$ are generated given the policy network $\pi_\theta(a_t|s_t)$.

From the source code, I saw that discrete actions get sampled from some probability distribution (which I assume to be discrete in this case) parameterized by the output probabilities generated by $\pi_\theta$ given the state $s_t$.

However, what I don't understand is how continuous actions are sampled/generated from the policy network. Are they also sampled from a (probably continuous) distribution? In that case, which type of distribution is used and which parameters are predicted by the policy network to parameterize said distribution?

Also, is there any official literature that I could cite which introduces the method by which PPO generates its action outputs?

I am trying to understand and reproduce the Proximal Policy Optimization (PPO) algorithm in detail. One thing that I find missing in the paper introducing the algorithm is how exactly actions $a_t$ are generated given the policy network $\pi_\theta(a_t|s_t)$.

From the source code, I saw that discrete actions get sampled from some probability distribution (which I assume to be discrete in this case) parameterized by the output probabilities generated by $\pi_\theta$ given the state $s_t$.

However, what I don't understand is how continuous actions are sampled/generated from the policy network. Are they also sampled from a (probably continuous) distribution? In that case, which type of distribution is used and which parameters are predicted by the policy network to parameterize said distribution?

Also, is there any official literature that I could cite which introduces the method by which PPO generates its action outputs?

I am trying to understand and reproduce the Proximal Policy Optimization (PPO) algorithm in detail. One thing that I find missing in the paper introducing the algorithm is how exactly actions $a_t$ are generated given the policy network $\pi_\theta(a_t|s_t)$.

From the source code, I saw that discrete actions get sampled from some probability distribution (which I assume to be discrete in this case) parameterized by the output probabilities generated by $\pi_\theta$ given the state $s_t$.

However, what I don't understand is how continuous actions are sampled/generated from the policy network. Are they also sampled from a (probably continuous) distribution? In that case, which type of distribution is used and which parameters are predicted by the policy network to parameterize said distribution?

Also, is there any official literature that I could cite which introduces the method by which PPO generates its action outputs?

Edit:

A question which I think is immediately related to the original question is how the probability ratio $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)}$ (as stated in the aforementioned paper) is calculated when working with continuous outputs generated by the policy network $\pi_\theta$.

Thus, given that $\pi_\theta$ predicts some mean $\mu$ and standard deviation $\sigma$ as outputs rather than a vector of discrete probability values, how is $r_t(\theta)$ calculated (since there are no probability value estimates $\pi_\theta(a_t|s_t)$ being generated any longer while $r_\theta$ is clearly called a probability ratio in the discrete case)?

I am trying to understand and reproduce the Proximal Policy Optimization (PPO) algorithm in detail. One thing that I find missing in the paper introducing the algorithm is how exactly actions $a_t$ are generated given the policy network $\pi_\theta(a_t|s_t)$.

From the source code, I saw that discrete actions get sampled from some probability distribution (which I assume to be discrete in this case) parameterized by the output probabilities generated by $\pi_\theta$ given state $s_t$.

However, what I don't understand is how continuous actions are sampled/generated from the policy network. Are they also sampled from a (probably continuous) distribution? In that case, which type of distribution is used and which parameters are predicted by the policy network to parameterize said distribution?

Also, is there any official literature that I could cite which introduces the method by which PPO generates its action outputs?

Thanks in advance!

I am trying to understand and reproduce the Proximal Policy Optimization (PPO) algorithm in detail. One thing that I find missing in the paper introducing the algorithm is how exactly actions $a_t$ are generated given the policy network $\pi_\theta(a_t|s_t)$.

From the source code, I saw that discrete actions get sampled from some probability distribution (which I assume to be discrete in this case) parameterized by the output probabilities generated by $\pi_\theta$ given the state $s_t$.

However, what I don't understand is how continuous actions are sampled/generated from the policy network. Are they also sampled from a (probably continuous) distribution? In that case, which type of distribution is used and which parameters are predicted by the policy network to parameterize said distribution?

Also, is there any official literature that I could cite which introduces the method by which PPO generates its action outputs?