1
$\begingroup$

I am relatively new to reinforcement learning, and I am trying to implement a reinforcement learning algorithm that can do continuous control in a custom environment. The state of the environment is composed of 5 sequential observations of 11 continuous values that represent cells (making the observation space 55 continuous values total), and the action is 11 continuous values representing multipliers of those cell totals.

After preliminary research, I decided to use Deep Deterministic Policy Gradient (DDPG) as my control algorithm because of its ability to deal with both discrete states and actions. However, most of the examples, including the one that I am basing my implementation off of, have only a single continuously valued action as the output. I have tried to naively change the agent network from outputting a single value to output a vector of values, but the agent does not improve as all, and the set of outputs seems to split into two groups near either the maximum value or the minimum value (I believe the tanh activation on the output has something to do with it) with the values in those groups changing in unison.

I have two questions about my problems.

  1. First, is it even possible to use DDPG for multi-dimensional continuous action spaces? My research leads me to believe it is, but I have not found any code examples to learn from and many of the papers I have read are near the limit of my understanding in this area.

  2. Second, why might my actor network be outputting values clustered near its max/min values, and why would the values in either cluster all be the same?

Again, I am fairly new to reinforcement learning, so any advice or recommendations would be greatly appreciated, thanks.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.