17
$\begingroup$

Trusted Region Policy Optimization (TRPO) and Proximal Policy Optimization (PPO) are two cutting edge policy gradients algorithms.

When using a single continuous action, normally, you would use some probability distribution (for example, Gaussian) for the loss function. The rough version is:

$$L(\theta) = \log(P(a_1)) A,$$

where $A$ is the advantage of rewards, $P(a_1)$ is characterized by $\mu$ and $\sigma^2$ that comes out of neural network like in the Pendulum environment here: https://github.com/leomzhong/DeepReinforcementLearningCourse/blob/69e573cd88faec7e9cf900da8eeef08c57dec0f0/hw4/main.py.

The problem is that I cannot find any paper on 2+ continuous actions using policy gradients (not actor-critic methods that use a different approach by transferring gradient from Q-function).

Do you know how to do this using TRPO for 2 continuous actions in LunarLander environment?

Is following approach correct for policy gradient loss function?

$$L(\theta) = (\log P(a_1) + \log P(a_2) )*A$$

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
8
$\begingroup$

As you has said, actions chosen by Actor-Critic typically come from a normal distribution and it is the agent's job to find the appropriate mean and standard deviation based on the the current state. In many cases this one distribution is enough because only 1 continuous action is required. However, as domains such as robotics become more integrated with AI, situations where 2 or more continuous actions are required are a growing problem.

There are 2 solutions to this problem: The first and most common is that for every continuous action, there is a separate agent learning its own 1-dimensional mean and standard deviation. Part of its state includes the actions of the other agents as well to give context of what the entire system is doing. We commonly do this in my lab and here is a paper which describes this approach with 3 actor-critic agents working together to move a robotic arm.

The second approach is to have one agent find a multivariate (usually normal) distribution of a policy. Although in theory, this approach could have a more concise policy distribution by "rotating" the distribution based on the co-variance matrix, it means that all of the values of the co-variance matrix must be learned as well. This increases the number of values that must be learned to have $n$ continuous outputs from $2n$ (mean and stddev), to $n+n^2$ ($n$ means and an $n \times n$ co-variance matrix). This drawback has made this approach not as popular in the literature.

This is a more general answer but should help you and others on their related problems.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Jaden thanks for great answer. 1. I tried multi-agent architecture, but it is not very efficient. Takes much longer to converge. 2. Now multivariate distribution seems obvious to me too, thank you. $\endgroup$
    – Evalds Urtans
    Sep 22, 2017 at 8:48
  • 1
    $\begingroup$ Depending on the application and architecture (if it is a deep net), you can have the agents share low level features and then have them branch off into their own value functions. Additionally, having 1 critic and multiple actors is also a way to increase the architecture. $\endgroup$
    – Jaden Travnik
    Sep 22, 2017 at 13:14
  • $\begingroup$ At the moment I would like to apply your suggestions to TRPO (just policy gradient methods), not actor-critic. I am not very confident in gradient transfer from critic to actor - in many implementations I have seen it looks like that it should not work even though it does converge. $\endgroup$
    – Evalds Urtans
    Sep 25, 2017 at 11:55
  • 1
    $\begingroup$ Sorry for this noob question: How is this applied in actor-critic methods (where the actor can perform multiple simultaneous continuous actions), where the actor has the policy function and gets trained by policy gradient method? @JadenTravnik Can you please explain that in the answer under a new heading? $\endgroup$
    – Gokul NC
    Feb 12, 2018 at 7:13

You must log in to answer this question.