Timeline for How are continuous actions sampled (or generated) from the policy network in PPO?
Current License: CC BY-SA 4.0
11 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| May 4, 2023 at 11:11 | comment | added | Yan King Yin | @DanielB. It seems that in the continuous case one has to use the re-parameterization trick or other methods to deal with probability distributions. I'm not sure if the re-parameterization trick allows multi-modal probability distributions to be represented. This seems to be a complicated topic. | |
| Dec 16, 2020 at 20:57 | vote | accept | Daniel B. | ||
| Dec 14, 2020 at 12:06 | comment | added | Daniel B. | In the continuous case, how would the probability $\pi_\theta(a_t|s_t)$ be computed since we don't predict probability vectors any longer, but unconstrained real numbers instead? Just asking because we still need this to be able to compute the probability ratio $r(\theta)$. | |
| Dec 13, 2020 at 13:28 | vote | accept | Daniel B. | ||
| Dec 16, 2020 at 17:58 | |||||
| Dec 13, 2020 at 10:56 | comment | added | Daniel B. | Ok, thank you very much! I think I am just subconsciously a bit biased towards believing that one always has to have a certain kind of output layer architecture, since I previously worked with Q-learning where it is of course pretty much dictated that, however the output layer looks like, it must be able to predict arbitrary Q-values (which of course makes the use certain activation functions etc impractical) and where the meaning of the output is clearly predetermined (to be Q-values), whereas the interpretation of outputs in PPO, TRPO... seems to be much more flexible. | |
| Dec 13, 2020 at 5:59 | comment | added | kaiwenw | @DanielB. exactly! :) the essence of REINFORCE, PPO, TRPO, Q-learning are the way the actors are updated, rather than a specific deep network architecture. For example, PPO/TRPO tries to stay in a "Trust Region", regardless of what policy architecture you choose. | |
| Dec 13, 2020 at 2:04 | comment | added | Daniel B. | So, after all there's no such thing like a fixed output layer architecture that a deep RL model would have to use in order to qualify as a PPO variant? It's really up to the researcher to decide how he/she want's a PPO variant's policy network's output to look like? Just double-checking one more time. :) | |
| Dec 12, 2020 at 21:19 | comment | added | kaiwenw | In the discrete case, you can do epsilon greedy, softmax, or anything really. As long as the density of your policy is differentiable, you can run PPO | |
| Dec 12, 2020 at 14:03 | comment | added | Daniel B. | Thanks for your answer! So, as I see it, action values are indeed sampled from some distribution in both the continuous and discrete case. In the continuous case, always(?) a Gaussian distribution is used of which the parameters are predicted. But how does the distribution look like for the discrete case? Is it indeed that in the discrete case, probabilities for sampling actions are only given by (possibly non-Gaussian-like distributed) probability vectors? And is there any nice paper that officially introduced these sampling schemes? | |
| Dec 12, 2020 at 12:05 | history | edited | nbro | CC BY-SA 4.0 |
deleted 1 character in body
|
| Dec 12, 2020 at 6:22 | history | answered | kaiwenw | CC BY-SA 4.0 |