What are the available exploration strategies for continuous action space scenarios in RL?

Question

I'm building a deep neural network to serve as the policy estimator in an actor-critic reinforcement learning algorithm for a continuing (not episodic) case. I'm trying to determine how to explore the action space. I have read through this text book by Sutton, and, in section 13.7, he gives one way to explore a continuous action space. In essence, you train the policy model to give a mean and standard deviation as an output, so you can sample a value from that Gaussian distribution to pick an action. This just seems like the continuous action-space equivalent of an $\epsilon$-greedy policy.

Are there other continuous action space exploration strategies I should consider?

I've been doing some research online and found some articles related to RL in robotics and found that the PoWER and PI^2 algorithms do something similar to what is in the textbook.

Are these, or other, algorithms "better" (obviously depends on the problem being solved) alternatives to what is listed in the textbook for continuous action-space problems?

I know that this question could have many answers, but I'm just looking for a reasonably short list of options that people have used in real applications that work.

Answer 1

I have not personally worked enough with continuous action spaces to be capable of confidently giving advise based on my own experience, but I can point you to likely relevant research (more recent than the research you already pointed to yourself):

---

The most common / "popular" area of research in recent years that involves RL and continuous action spaces uses robot / physics simulators such as MuJoCo. Some examples:

• Asynchronous Methods for Deep Reinforcement Learning mentions using an entropy cost term in the loss function used for training to encourage exploration (see Supplementary Material after References).
• Parameter Space Noise for Exploration describes the idea of adding noise directly to the learned parameters on the Neural Network. That way it basically always predicts a different action to be "optimal" (due to the noise), and therefore a "greedy" policy based on the noisy parameters will in fact have exploration and not be fully greedy.
• DeepMind Control Suite is a somewhat recent paper that proposes a suite of benchmarks for continuous control problems. It is filled with references to relevant papers describing all kinds of algorithms, and hopefully every single one of those papers would also describe how they perform exploration.

---

Recently, at the ICML 2018 conference, there was a complete workshop dedicated to Exploration in Reinforcement Learning. Here is the list of accepted papers at this workshop. Note that it's about Exploration in RL, not only about continuous action spaces, so there might be papers in there that are only applicable to discrete action spaces. Nevertheless, I would be highly surprised if there's nothing in there that's relevant.

Answer 2

Firstly, note that the Gaussian policies you describe are not equivalent to $\epsilon$-greedy, mainly for this reason: for a fixed policy, the policy's variance in the Gaussian case depends on the state, while in the $\epsilon$-greedy case it does not. Right off the bat, the Gaussian policy should achieve less regret than $\epsilon$-greedy.

Other approaches to exploration in continuous action spaces include :

• Parameterizing the policy differently. You're not limited to gaussians, rather any parameterizable distribution (particularly those that can be reparameterized according to the reparameterization trick) will do.
• Using an entropy bonus. You can subtract your policy's entropy in the expression for your loss function, which helps prevent your policy from becoming "too deterministic" before the agent learns the environment sufficiently.
• Surprise/curiosity based methods. From this I mean methods that do reward shaping based on some measure of uncertainty in the policy -- at each transition, this measure of uncertainty is added to the reward. See "Exploration by Random Network Distillation" for example.
• Maximum Entropy methods. These have slightly different objectives than standard RL that also emphasize policy entropy, so they should promote exploration. See SAC for example.
• Use deterministic policy gradients. Then you can literally apply $\epsilon$-greedy if you want, or simply add noise to the output of the policy. See TD3 for example.

I doubt this is an exhaustive list, but I hope it helps.