I'm building a deep neural network to serve as the policy estimator in an Actor-Critic reinforcement learning algorithm for a continuing (not episodic) case. I'm trying to determine how to explore the action space. I have read through this text book by Sutton and in section 13.7 he gives one way to explore a continuous action space. In essence you train the policy model to give a mean and standard deviation as an output so you can sample a value from that Gaussian distribution to pick an action. This just seems like the continuous action-space equivalent of an epsilon greedy policy.

Also, are there other continuous action space exploration strategies I should consider? I've been doing some research online and found some articles related to RL in robotics and found that the PoWER and PI^2 algorithms do something similar to what is in the textbook. Are these, or other, algorithms "better" (obviously depends on the problem being solved) alternatives to what is listed in the textbook for continuous action-space problems?

I know that this question could have many answers, but I'm just looking for a reasonably short list of options that people have used in real applications that work.

  • $\begingroup$ If one of the actions is the bullet that killed J. F. Kennedy, a trajectory of a meteor toward earth, a knock out in a fight, or diamond in the sand, the Gaussian distribution wouldn't work. Human sensory learning does not require normalcy, and the direction of research should assume from the onset that the distribution is unknown. $\endgroup$ – FauChristian Aug 4 '18 at 2:33

I have not personally worked enough with continuous action spaces to be capable of confidently giving advise based on my own experience, but I can point you to likely relevant research (more recent than the research you already pointed to yourself):

The most common / "popular" area of research in recent years that involves RL and continuous action spaces uses robot / physics simulators such as MuJoCo. Some examples:

  • Asynchronous Methods for Deep Reinforcement Learning mentions using an entropy cost term in the loss function used for training to encourage exploration (see Supplementary Material after References).
  • Parameter Space Noise for Exploration describes the idea of adding noise directly to the learned parameters on the Neural Network. That way it basically always predicts a different action to be "optimal" (due to the noise), and therefore a "greedy" policy based on the noisy parameters will in fact have exploration and not be fully greedy.
  • DeepMind Control Suite is a somewhat recent paper that proposes a suite of benchmarks for continuous control problems. It is filled with references to relevant papers describing all kinds of algorithms, and hopefully every single one of those papers would also describe how they perform exploration.

Recently, at the ICML 2018 conference, there was a complete workshop dedicated to Exploration in Reinforcement Learning. Here is the list of accepted papers at this workshop. Note that it's about Exploration in RL, not only about continuous action spaces, so there might be papers in there that are only applicable to discrete action spaces. Nevertheless, I would be highly surprised if there's nothing in there that's relevant.

| improve this answer | |
  • 1
    $\begingroup$ Thanks for the list of places to look. I like the idea of doing exploration by perturbing the weights of the Neural Network. This type of method can be used in discrete or continuous action spaces and has the added benefit of allowing random logic (inside the NN) to be tried for a while, then shifted back if it doesn't help. A method similar to the one cited above can be found in NOISY NETWORKS FOR EXPLORATION by the Google Deep Mind team. $\endgroup$ – Jed Aug 9 '18 at 21:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.