There is a recent paper: Continuous-Discrete Reinforcement Learning for
Hybrid Control in Robotics published by DeepMind that aims to solve this problem, as stated in the abstract:
Many real-world control problems involve both discrete decision variables – such as the choice of control modes, gear switching or digital outputs – as
well as continuous decision variables – such as velocity setpoints, control gains or
analogue outputs. However, when defining the corresponding optimal control or reinforcement learning problem, it is commonly approximated with fully continuous
or fully discrete action spaces. These simplifications aim at tailoring the problem
to a particular algorithm or solver which may only support one type of action
space. Alternatively, expert heuristics are used to remove discrete actions from
an otherwise continuous space. In contrast, we propose to treat hybrid problems
in their ‘native’ form by solving them with hybrid reinforcement learning, which
optimizes for discrete and continuous actions simultaneously.
The idea is that they use a hybrid policy that uses a Gaussian distribution for the continuous decision variables and a categorical distribution for the discrete decision variables. Then, they extend the Maximum a Posteriori Policy Optimisation (MPO) algorithm (also by DeepMind) to allow it to handle hybrid policies.
Here is a video showing how they used the resulting hybrid MPO policy in a robotics task, where in addition to the continuous actions, the robot can choose a discrete action which is the control mode to be used (coarse vs. fine).