# How can policy gradients be applied in the case of multiple continuous actions?

Trusted Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) are two cutting edge policy gradients algorithms.

When using a single continuous action, normally, you would use some probability distribution (for example, Gaussian) for the loss function. The rough version is:

$$L(\theta) = \log(P(a_1)) A,$$

where $$A$$ is the advantage of rewards, $$P(a_1)$$ is characterized by $$\mu$$ and $$\sigma^2$$ that comes out of neural network like in the Pendulum environment here: https://github.com/leomzhong/DeepReinforcementLearningCourse/blob/69e573cd88faec7e9cf900da8eeef08c57dec0f0/hw4/main.py.

The problem is that I cannot find any paper on 2+ continuous actions using policy gradients (not actor-critic methods that use a different approach by transferring gradient from Q-function).

Do you know how to do this using TRPO for 2 continuous actions in LunarLander environment?

Is following approach correct for policy gradient loss function?

$$L(\theta) = (\log P(a_1) + \log P(a_2) )*A$$

The second approach is to have one agent find a multivariate (usually normal) distribution of a policy. Although in theory, this approach could have a more concise policy distribution by "rotating" the distribution based on the co-variance matrix, it means that all of the values of the co-variance matrix must be learned as well. This increases the number of values that must be learned to have $n$ continuous outputs from $2n$ (mean and stddev), to $n+n^2$ ($n$ means and an $n \times n$ co-variance matrix). This drawback has made this approach not as popular in the literature.