Trusted Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) are two cutting edge policy gradients algorithms.
When using a single continuous action, normally, you would use some probability distribution (for example, Gaussian) for the loss function. The rough version is:
$$L(\theta) = \log(P(a_1)) A,$$
where $A$ is the advantage of rewards, $P(a_1)$ is characterized by $\mu$ and $\sigma^2$ that comes out of neural network like in the Pendulum environment here: https://github.com/leomzhong/DeepReinforcementLearningCourse/blob/69e573cd88faec7e9cf900da8eeef08c57dec0f0/hw4/main.py.
The problem is that I cannot find any paper on 2+ continuous actions using policy gradients (not actor-critic methods that use a different approach by transferring gradient from Q-function).
Do you know how to do this using TRPO for 2 continuous actions in LunarLander environment?
Is following approach correct for policy gradient loss function?
$$L(\theta) = (\log P(a_1) + \log P(a_2) )*A$$