# Policy gradients for multiple continuous actions

Question is regarding Deep Reinforcement Learning using Policy Gradients.

Cutting edge policy gradients algorithms are TRPO (Trusted Region Policy Optimization) and PPO (Proximal Policy Optimization).

When using single continuous action then normally you would use some random distribution (for example Gaussian) for the loss function. The rough version is:

$L(\theta) = log(P(a_1)) * A$

Where $A$ is the advantage of rewards $P(a_1)$ is characterized by $\mu$ and $\sigma^2$ that comes out of neural network like in Pendulum environment here: https://github.com/leomzhong/DeepReinforcementLearningCourse/blob/69e573cd88faec7e9cf900da8eeef08c57dec0f0/hw4/main.py

Problem is that I cannot find any paper on 2+ continuous actions using policy gradients (not Actor-critic methods that use a different approach by transferring gradient from Q-Function).

Do you know how to do this using TRPO for 2 continuous actions in LunarLander environment? https://gym.openai.com/envs/LunarLanderContinuous-v2/

Is following approach correct for policy gradient loss function?

$L(\theta) = (log(P(a_1))+log(P(a_2)))*A$

• I kindly request you to revise your question,a little bit and post it here Cross Validated – quintumnia Sep 21 '17 at 16:28
• ok, next time I will post there, but below answer is great – Evalds Urtans Sep 22 '17 at 8:50
• Welcome to AI! We definitely allow conceptual queries on AI methods, so thank you for asking this here. (For more specifically implementation-oriented questions, Cross Validated or Data Science are definitely preferred.) – DukeZhou Sep 22 '17 at 21:15
• You should add LaTex math parser as in Data Science exchange, especially if you are focused on theory aspects – Evalds Urtans Sep 25 '17 at 11:56
• What about ur loss function ? – Shamane Siriwardhana Dec 21 '18 at 3:51

The second approach is to have one agent find a multivariate (usually normal) distribution of a policy. Although in theory, this approach could have a more concise policy distribution by "rotating" the distribution based on the co-variance matrix, it means that all of the values of the co-variance matrix must be learned as well. This increases the number of values that must be learned to have $n$ continuous outputs from $2n$ (mean and stddev), to $n+n^2$ ($n$ means and an $n \times n$ co-variance matrix). This drawback has made this approach not as popular in the literature.