1
$\begingroup$

In supervised or unsupervised learning, it is advised to reduce the dimensionality due to the curse of dimensionality in general.

Is this also generally advisable for the action space of reinforcement learning?

As far as I understand (and inspired by the answer here Is it possible to tell the Reinforcement Learning agent some rules directly without any constraints), you can reduce the dimensionality of the action space always to 1, meaning that you solely have 1 action. This can be done by using a mapping (e.g. in the step function of Open AI Gym).

Let's have a look at an example: We have a heating device that can heat 3 storages and we have a discrete action variable for all of them with 11 steps [0.0, 0.1, 0.2, ..., 1.0]. So we have

  • action_heatStorage1: [0.0, 0.1, 0.2, ..., 1.0]
  • action_heatStorage2: [0.0, 0.1, 0.2, ..., 1.0]
  • action_heatStorage3: [0.0, 0.1, 0.2, ..., 1.0]

In this case, we would have a 3-dimensional action space

  • action = [action_heatStorage1, action_heatStorage2, action_heatStorage3]

However, it is also possible to combine the 3 actions into 1 action variable "action_combined" of the size [11 * 11 * 11=1331] by just using a mapping of this one action into the separate 3 actions. For example like this:

  • action_combined = 0 --> action_heatStorage1 =0, action_heatStorage2 =0, action_heatStorage3 =0
  • action_combined = 1 --> action_heatStorage1 =0.1, action_heatStorage2 =0, action_heatStorage3 =0
  • action_combined = 2 --> action_heatStorage1 =0.2, action_heatStorage2 =0, action_heatStorage3 =0

...

  • action_combined = 1331 --> action_heatStorage1 =1.0, action_heatStorage2 =1.0, action_heatStorage3 =1.0

Is it generally advisable to reduce the dimensionality of the action space (Option 2), or to use multidimensional action variables directly (Option 1)?

I know that the is most probably not an answer that is valid for all problems. But, as I am relatively new to reinforcement learning, I would like to know whether in the theory of reinforcement learning there is a general recommendation to do something like this or not or whether this question can't be answered in general as it is something that totally depends on the application and should be tested for each application individually?

Reminder: I have already received a good answer. Still, I would like to remind you on this question to maybe hear also the opinion and experience of others regarding this topic.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Since the question may not be answered unambiguously in general, I will use the given example as a guide. As you correctly write, a large dimensionality leads to a very large solution space because of the curse of dimensions.

However, whether one restricts this solution space by allowing only discrete values $[0.0, 0.1, 0.2, ..., 1.0]$ for the three actions or by using the resulting combinations as a list of options (one dimension) makes no difference with respect to solution space coverage. Depending on which inputs the model receives, however, it would be conceivable that the separate variables (option 1) could be more easily assigned and learned, e.g. if there are sensors for the state, temperature, etc. of each of the 3 heat storages as inputs. Option 2 would still require the model to implicitly learn one-dimensional coding to target specific heat storages.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thanks dexteritas for your answer. So you would tend not to reduce the dimensionality but just use the 3 action variables separately? One big advantage of transforming them into a one dimensional action space is that you could map this one action to only valid options in a control problem and thus tell the agent some rules directly independant from the reward function (see ai.stackexchange.com/questions/32042/…). But I also understand your point. Most probably it will depend on the application. $\endgroup$
    – PeterBe
    Nov 15, 2021 at 15:57
  • $\begingroup$ Yes, in the given example, I would use the 3 actions variables separately. Of course, it depends on the problem, e.g. if you have only 2 or 3 states for each of the 3 actions a combination could be more useful. Depending on the complexity of the control problem, it could become difficult to identify and exclude all forbidden cases and be easier to use a constraint function. Especially, if the validity not only depends on the produces outputs, but also on the current state (so a output could be valid but is not valid for all cases). However, it depends ;) $\endgroup$
    – dexteritas
    Nov 15, 2021 at 16:19
  • $\begingroup$ Thanks for your comment dexteritas. What do you mean by "constraint function"? A function that is influecing the reward in a negative way if a constraint violation has occured or a super-ordinate controller that can overrule the inital actions of the RL agent? $\endgroup$
    – PeterBe
    Nov 15, 2021 at 16:22
  • $\begingroup$ You have a constrained optimization problem. There are different approaches for that. Giving a negative reward would be a penalty function. I think it is useful, if the agent can learn what is valid/allowed. You could use e.g. penalty function in combination with super-ordinate controller to ensure hard constraints are not violated. But that could be a separate question. $\endgroup$
    – dexteritas
    Nov 15, 2021 at 16:32
  • 1
    $\begingroup$ As nobody other than you answered my question, I have now also accepted your answer (I had upvoted it before). I appreciate it. $\endgroup$
    – PeterBe
    Nov 26, 2021 at 8:22

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .