# Policy gradients for multiple continuous actions

Question is regarding Deep Reinforcement Learning using Policy Gradients.

Cutting edge policy gradients algorithms are TRPO (Trusted Region Policy Optimization) and PPO (Proximal Policy Optimization).

When using single continuous action then normally you would use some random distribution (for example Gaussian) for the loss function. The rough version is:

$L(\theta) = log(P(a_1)) * A$

Where $A$ is the advantage of rewards $P(a_1)$ is characterized by $\mu$ and $\sigma^2$ that comes out of neural network like in Pendulum environment here: https://github.com/leomzhong/DeepReinforcementLearningCourse/blob/69e573cd88faec7e9cf900da8eeef08c57dec0f0/hw4/main.py

Problem is that I cannot find any paper on 2+ continuous actions using policy gradients (not Actor-critic methods that use a different approach by transferring gradient from Q-Function).

Do you know how to do this using TRPO for 2 continuous actions in LunarLander environment? https://gym.openai.com/envs/LunarLanderContinuous-v2/

Is following approach correct for policy gradient loss function?

$L(\theta) = (log(P(a_1))+log(P(a_2)))*A$

• I kindly request you to revise your question,a little bit and post it here Cross Validated – quintumnia Sep 21 '17 at 16:28
• ok, next time I will post there, but below answer is great – Evalds Urtans Sep 22 '17 at 8:50
• Welcome to AI! We definitely allow conceptual queries on AI methods, so thank you for asking this here. (For more specifically implementation-oriented questions, Cross Validated or Data Science are definitely preferred.) – DukeZhou Sep 22 '17 at 21:15
• You should add LaTex math parser as in Data Science exchange, especially if you are focused on theory aspects – Evalds Urtans Sep 25 '17 at 11:56
• What about ur loss function ? – Shamane Siriwardhana Dec 21 '18 at 3:51

## 1 Answer

As you has said, actions chosen by Actor-Critic typically come from a normal distribution and it is the agent's job to find the appropriate mean and standard deviation based on the the current state. In many cases this one distribution is enough because only 1 continuous action is required. However, as domains such as robotics become more integrated with AI, situations where 2 or more continuous actions are required are a growing problem.

There are 2 solutions to this problem: The first and most common is that for every continuous action, there is a separate agent learning its own 1-dimensional mean and standard deviation. Part of its state includes the actions of the other agents as well to give context of what the entire system is doing. We commonly do this in my lab and here is a paper which describes this approach with 3 actor-critic agents working together to move a robotic arm.

The second approach is to have one agent find a multivariate (usually normal) distribution of a policy. Although in theory, this approach could have a more concise policy distribution by "rotating" the distribution based on the co-variance matrix, it means that all of the values of the co-variance matrix must be learned as well. This increases the number of values that must be learned to have $n$ continuous outputs from $2n$ (mean and stddev), to $n+n^2$ ($n$ means and an $n \times n$ co-variance matrix). This drawback has made this approach not as popular in the literature.

This is a more general answer but should help you and others on their related problems.

• Jaden thanks for great answer. 1. I tried multi-agent architecture, but it is not very efficient. Takes much longer to converge. 2. Now multivariate distribution seems obvious to me too, thank you. – Evalds Urtans Sep 22 '17 at 8:48
• Depending on the application and architecture (if it is a deep net), you can have the agents share low level features and then have them branch off into their own value functions. Additionally, having 1 critic and multiple actors is also a way to increase the architecture. – Jaden Travnik Sep 22 '17 at 13:14
• At the moment I would like to apply your suggestions to TRPO (just policy gradient methods), not actor-critic. I am not very confident in gradient transfer from critic to actor - in many implementations I have seen it looks like that it should not work even though it does converge. – Evalds Urtans Sep 25 '17 at 11:55
• Sorry for this noob question: How is this applied in actor-critic methods (where the actor can perform multiple simultaneous continuous actions), where the actor has the policy function and gets trained by policy gradient method? @JadenTravnik Can you please explain that in the answer under a new heading? – Gokul NC Feb 12 '18 at 7:13