I'm interested in trying out Q-learning to solve a problem where I already have a simulation of the environment that can run at about 100,000 fps or steps/sec. Its also continuous with no terminal states.

The estimated state space should be no more than 100,000. Most state can take integer values from 0 to 200.

As for the action space, I am unsure if it should be 10, or if the action space should be 1000 (2**10). Basically there are 10 possible individual actions, but the actions can be pressed in all sorts of combinations, like action 1, 2, 3, and 5 can be taken at the same time.

In this case, can tabular method still work fine? If so, are there any advantages of using a neural network, like DQN or PPO?


1 Answer 1


If I read correctly, your RL action space is a Multi-Discrete one, where each action is independent of each other and can be used simultaneously (like controller or keyboard), which is supported by Stable-baselines3 PPO algorithm (but not DQN by the way). Choosing an action space is called "action space shaping", and it is crucial for the performance of a RL algorithm.

Luckily, this paper studies exactly just that, although it just studies PPO - but PPO is probably the most popular RL algorithm out there. It benchmarks 11 games over a wide range of action space shaping - like in your case, whether it should be 10 independent actions, or 1000 one-hot actions. The conclusion is that using a Multi Discrete action is a better choice:

Avoid turning multi-discrete actions into a single discrete action and limit the number of choices per discrete action.

So in your case, it is better to go a Multi Discrete with 10 actions. I highly recommend reading it for more insights.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .