Using DDPG for control in multi-dimensional continuous action space?

Question

I am relatively new to reinforcement learning, and I am trying to implement a reinforcement learning algorithm that can do continuous control in a custom environment. The state of the environment is composed of 5 sequential observations of 11 continuous values that represent cells (making the observation space 55 continuous values total), and the action is 11 continuous values representing multipliers of those cell totals.

After preliminary research, I decided to use Deep Deterministic Policy Gradient (DDPG) as my control algorithm because of its ability to deal with both discrete states and actions. However, most of the examples, including the one that I am basing my implementation off of, have only a single continuously valued action as the output. I have tried to naively change the agent network from outputting a single value to output a vector of values, but the agent does not improve as all, and the set of outputs seems to split into two groups near either the maximum value or the minimum value (I believe the tanh activation on the output has something to do with it) with the values in those groups changing in unison.

I have two questions about my problems.

1. First, is it even possible to use DDPG for multi-dimensional continuous action spaces? My research leads me to believe it is, but I have not found any code examples to learn from and many of the papers I have read are near the limit of my understanding in this area.

2. Second, why might my actor network be outputting values clustered near its max/min values, and why would the values in either cluster all be the same?

Again, I am fairly new to reinforcement learning, so any advice or recommendations would be greatly appreciated, thanks.

Answer 1

First, is it even possible to use DDPG for multi-dimensional continuous action spaces?

Yes, DDPG was primarily developed to deal with continuous action space you can find out more here, here and here.

I have not found any code examples to learn from and many of the papers I have read are near the limit of my understanding in this area.

You can check kerasRL ddpg implementation it is quite easy to understand and can solve your problem of a model with multiple outputs..

why might my actor network be outputting values clustered near its max/min values, and why would the values in either cluster all be the same?

I think it is just a convergence problem try out small learning rates and/or small target_model_update param.

If this doesn't work for you check TRPO and/or PPO..