# Is there a difference in the architecture of deep reinforcement learning when multiple actions are performed instead of a single action?

I've built a deep deterministic policy gradient reinforcement learning agent to be able to handle any games / tasks that have only one action. However, the agent seems to fail horribly when there are two or more actions. I tried to look online for any examples of somebody implementing DDPG on a multiple action system, but people mostly applied it to the pendulum problem, which is a single action problem.

For my current system, it is a 3 state, 2 continuous control actions system (One is to adjust the temperature of the system, the other one adjusts a mechanical position, both are continuous). However, I froze the second continuous action to be the optimal action all the time. So RL only has to manipulate one action. It solves within 30 episodes. However, the moment I allow the RL to try both continuous actions, it doesn't even converge after 1000 episodes. In fact, it diverges aggressively. The output of the actor network seems to always be the max action, possibly because I am using a tanh activation for the actor to provide output constraint. I added a penalty to large actions, but it does not seem to work for the 2 continuous control action case.

For my exploratory noise, I used Ornstein-Ulhenbeck noise, with means adjusted for the two different continuous actions. The mean of the noise is 10% of the mean of the action.

Is there any massive difference between single action and multiple action DDPG? I changed the reward function to take into account both actions, have tried making a bigger network, tried priority replay, etc., but it appears I am missing something. Does anyone here have any experience building a multiple action DDPG and could give me some pointers?

• Technically, the difference here is between actions in (some subset of) $\mathbb{R}$ and $\mathbb{R}^n$, not between 1 or more "actions". In other words, you have an action space here that might have multiple dimensions, and something is going wrong for your agent when there are 2 or more dimensions. In RL, when something described as "having 2 actions" this is usually an enumeration - i.e. the agent can take action A or action B, and there are no quantities involved. – Neil Slater Aug 25 '18 at 7:09
• Hi Neil, thanks for the reply. Yes, for classic RL, it agents' actions are indeed discrete. However, in 2015, Lilicrap published a paper called "continuous control with deep reinforcement learning", and then in 2017, the TRPO and PPO algorithms were designed to allow agents to perform multiple continuous actions. So you are correct about my action being in a high dimension space. In my research, I am comparing model predictive control using trajectory optimization vs AI-based control. Usually, in robotics and mechatronics, robots move multiple pieces. I am trying to achieve that with RL. – Rui Nian Aug 26 '18 at 4:41
• I suggest you edit a more accurate description of your RL problem to replace the sentence "For my current system, it is a 3 state, 2 action system." - because that is not how it would be described in any literature. May also be worth explaining how you have adjusted the exploration function ("actor noise"), as a mistake there would be key. – Neil Slater Aug 26 '18 at 9:16
• Done! I will also try different exploratory noise means to see if it helps. – Rui Nian Aug 27 '18 at 3:46
• Thanks. I was wondering if you had somehow failed to adjust for different scales of the two axes of action, but it doesn't look like it. I cannot really tell what is wrong. However, I would not personally expect DDPG to be quite so fragile when scaling up from one to two dimensions of action, so I'd still suspect something about your implementation - I just don't know what it could be. – Neil Slater Aug 28 '18 at 7:41

First Stated Question

Is there a difference in the architecture of deep reinforcement learning when multiple actions are performed instead of a single action?

The way the question is phrased implies that the query is about a discrete implication, that an architectural change is an imperative. It is not, since an action may be comprised of multiple actions, whether or not there are sequencing dependencies on the component actions. In the case of the control of two physical properties, the control space has two degrees of freedom. That they are controlled using discrete corrections leads to a hybrid of continuous and discrete mathematics, which is commonplace in control.

From the body and the comments it is likely that the question author is privy to these facts. One of the two main questions described is whether gains can be achieved with more sophisticated process topology or other strategic applications of expectation and probability distribution math. Such gains might be achievable.

• Faster response (temporal accuracy)
• Accuracy in objective tracking (independent of time)
• Tracking reliability (no gross loss of synchronization due to signal saturation or clipping)
• Risk aversion (steering clear of irretrievable loss in sparsely or weakly characterized path spaces)

In the case of temperature and position, further topological sophistication is not likely.

Longer Term Goal of Research

Later along the research path, topological changes in process and signal flow (early in the systems architecture development) will probably be effective in improving system quality. This is likely in light of the stated intention to produce a smart learning controller using the best from multiple conceptual sources.

• Deterministic policy gradient reinforcement learning agent, the proof of concept of which is converging in 30 episodes with one degree of freedom, position
• Lilicrap's Continuous control with deep reinforcement learning, 2015
• TRPO and PPO algorithms agents to perform multiple continuous actions, 2017
• Tesla megafactory
• Predictive control using trajectory optimization
• Automated, progressive model development

Whether there is an intersection point of all six that benefits from the contribution of each is unlikely, but a reasonable hypothesis to test.

Immediate Concern

The description of the current issue is not closely related to the first stated question or the ultimate goal but rather an anomaly in the current proof of concept.

That adding a second degree of freedom, temperature, "Fail[s] horribly [and] diverges aggressively," before reaching 1,000 episodes is indeed an anomaly. The injection of -20 dB of Ornstein-Ulhenbeck noise as measured by mean amplitude (10%) to avoid search pitfalls is unlikely to be related to

Is there any massive difference between single [degrees of freedom] and multiple [degrees of freedom in] DDPG?

Only if the person extending the software is not adept with multivariate calculus.

The remedies tried don't seem to be producing results, which is not surprising since none have to do with a likely root cause.

• Reward function aggregating actions
• Bigger network
• Priority replay
• Activation of tanh
• Penalty to large actions

The sixth thing mentioned may be more likely to remedy the divergence.

• New interpretations of actions and rewards

The particular anomaly described, albeit without much detail, points to some common causes of unexpected gross divergence.

• Mishandling of a minus sign during performance of the calculus or associated algebra
• A flaw in a partial derivative
• Using only the diagonal of the Jacobian, or the dismissal of some other pattern within the Jacobian in its application to corrective signalling or predictive quantification
• Hi Douglas, thanks for the reply. You answer was certainly very helpful. The issue actually arose from integral wind-up states. Currently, do you know of any methods that can handle integral wind-up states? Thanks again for your answer! – Rui Nian Nov 29 '18 at 14:53
• signal.uu.se/Publications/pdf/a032.pdf – Douglas Daseeco Nov 29 '18 at 17:07