In supervised or unsupervised learning, it is advised to reduce the dimensionality due to the curse of dimensionality in general.
Is this also generally advisable for the action space of reinforcement learning?
As far as I understand (and inspired by the answer here Is it possible to tell the Reinforcement Learning agent some rules directly without any constraints), you can reduce the dimensionality of the action space always to 1, meaning that you solely have 1 action. This can be done by using a mapping (e.g. in the step function of Open AI Gym).
Let's have a look at an example: We have a heating device that can heat 3 storages and we have a discrete action variable for all of them with 11 steps [0.0, 0.1, 0.2, ..., 1.0]. So we have
- action_heatStorage1: [0.0, 0.1, 0.2, ..., 1.0]
- action_heatStorage2: [0.0, 0.1, 0.2, ..., 1.0]
- action_heatStorage3: [0.0, 0.1, 0.2, ..., 1.0]
In this case, we would have a 3-dimensional action space
- action = [action_heatStorage1, action_heatStorage2, action_heatStorage3]
However, it is also possible to combine the 3 actions into 1 action variable "action_combined" of the size [11 * 11 * 11=1331] by just using a mapping of this one action into the separate 3 actions. For example like this:
- action_combined = 0 --> action_heatStorage1 =0, action_heatStorage2 =0, action_heatStorage3 =0
- action_combined = 1 --> action_heatStorage1 =0.1, action_heatStorage2 =0, action_heatStorage3 =0
- action_combined = 2 --> action_heatStorage1 =0.2, action_heatStorage2 =0, action_heatStorage3 =0
- action_combined = 1331 --> action_heatStorage1 =1.0, action_heatStorage2 =1.0, action_heatStorage3 =1.0
Is it generally advisable to reduce the dimensionality of the action space (Option 2), or to use multidimensional action variables directly (Option 1)?
I know that the is most probably not an answer that is valid for all problems. But, as I am relatively new to reinforcement learning, I would like to know whether in the theory of reinforcement learning there is a general recommendation to do something like this or not or whether this question can't be answered in general as it is something that totally depends on the application and should be tested for each application individually?
Reminder: I have already received a good answer. Still, I would like to remind you on this question to maybe hear also the opinion and experience of others regarding this topic.