Learning policy where action involves discrete and continuous parameters

Question

Typically it seems like reinforcement learning involves learning over either a discrete or a continuous action space. An example might be choosing from a set of pre-defined game actions in Gym Retro or learning the right engine force to apply in Continuous Mountain Car; some popular approaches for these problems are deep Q-learning for the former and actor-critic methods for the latter.

What about in the case where a single action involves picking both a discrete and a continuous parameter? For example, when choosing the type (discrete), pixel grid location (discrete), and angular orientation (continuous) of a shape from a given set to place on a grid and optimize for some reward. Is there a well-established approach for learning a policy to make both types of decisions at once?

Answer 1

There is a recent paper: Continuous-Discrete Reinforcement Learning for Hybrid Control in Robotics published by DeepMind that aims to solve this problem, as stated in the abstract:

Many real-world control problems involve both discrete decision variables – such as the choice of control modes, gear switching or digital outputs – as well as continuous decision variables – such as velocity setpoints, control gains or analogue outputs. However, when defining the corresponding optimal control or reinforcement learning problem, it is commonly approximated with fully continuous or fully discrete action spaces. These simplifications aim at tailoring the problem to a particular algorithm or solver which may only support one type of action space. Alternatively, expert heuristics are used to remove discrete actions from an otherwise continuous space. In contrast, we propose to treat hybrid problems in their ‘native’ form by solving them with hybrid reinforcement learning, which optimizes for discrete and continuous actions simultaneously.

The idea is that they use a hybrid policy that uses a Gaussian distribution for the continuous decision variables and a categorical distribution for the discrete decision variables. Then, they extend the Maximum a Posteriori Policy Optimisation (MPO) algorithm (also by DeepMind) to allow it to handle hybrid policies.

Here is a video showing how they used the resulting hybrid MPO policy in a robotics task, where in addition to the continuous actions, the robot can choose a discrete action which is the control mode to be used (coarse vs. fine).