First Stated Question
Is there a difference in the architecture of deep reinforcement learning when multiple actions are performed instead of a single action?
The way the question is phrased implies that the query is about a discrete implication, that an architectural change is an imperative. It is not, since an action may be comprised of multiple actions, whether or not there are sequencing dependencies on the component actions. In the case of the control of two physical properties, the control space has two degrees of freedom. That they are controlled using discrete corrections leads to a hybrid of continuous and discrete mathematics, which is commonplace in control.
From the body and the comments it is likely that the question author is privy to these facts. One of the two main questions described is whether gains can be achieved with more sophisticated process topology or other strategic applications of expectation and probability distribution math. Such gains might be achievable.
- Faster response (temporal accuracy)
- Accuracy in objective tracking (independent of time)
- Tracking reliability (no gross loss of synchronization due to signal saturation or clipping)
- Risk aversion (steering clear of irretrievable loss in sparsely or weakly characterized path spaces)
In the case of temperature and position, further topological sophistication is not likely.
Longer Term Goal of Research
Later along the research path, topological changes in process and signal flow (early in the systems architecture development) will probably be effective in improving system quality. This is likely in light of the stated intention to produce a smart learning controller using the best from multiple conceptual sources.
- Deterministic policy gradient reinforcement learning agent, the proof of concept of which is converging in 30 episodes with one degree of freedom, position
- Lilicrap's Continuous control with deep reinforcement learning, 2015
- TRPO and PPO algorithms agents to perform multiple continuous actions, 2017
- Tesla megafactory
- Predictive control using trajectory optimization
- Automated, progressive model development
Whether there is an intersection point of all six that benefits from the contribution of each is unlikely, but a reasonable hypothesis to test.
The description of the current issue is not closely related to the first stated question or the ultimate goal but rather an anomaly in the current proof of concept.
That adding a second degree of freedom, temperature, "Fail[s] horribly [and] diverges aggressively," before reaching 1,000 episodes is indeed an anomaly. The injection of -20 dB of Ornstein-Ulhenbeck noise as measured by mean amplitude (10%) to avoid search pitfalls is unlikely to be related to
Is there any massive difference between single [degrees of freedom] and multiple [degrees of freedom in] DDPG?
Only if the person extending the software is not adept with multivariate calculus.
The remedies tried don't seem to be producing results, which is not surprising since none have to do with a likely root cause.
- Reward function aggregating actions
- Bigger network
- Priority replay
- Activation of tanh
- Penalty to large actions
The sixth thing mentioned may be more likely to remedy the divergence.
- New interpretations of actions and rewards
The particular anomaly described, albeit without much detail, points to some common causes of unexpected gross divergence.
- Mishandling of a minus sign during performance of the calculus or associated algebra
- A flaw in a partial derivative
- Using only the diagonal of the Jacobian, or the dismissal of some other pattern within the Jacobian in its application to corrective signalling or predictive quantification