I want to use Reinforcement Learning to optimize the distribution of energy for a peak shaving problem given by a thermodynamical simulation. However, I am not sure how to proceed as the action space is the only thing that really matters, in this sense:

  • The action space is a 288x66 matrix of real numbers between 0 and 1. The output of the simulation and therefore my reward depend solely on the distribution of this matrix.
  • The state space is therefore absent, as the only thing that matters is the matrix on which I have total control. At this stage of the simulation, no other variables are taken into consideration.

I am not sure if this problem falls into the tabular RL or it requires approximation. In this case, I was thinking about using a Policy gradient algorithm for figuring out the best distribution of the 288x66 matrix; however, I do not know how to behave with the "absence" of the state space. Instead of a tuple I would just have , is this even a RL-approachable problem? If not, how can I reshape it to make it solvable with RL techniques?

  • $\begingroup$ The question sounds, if the problem is completely new and was never described in the literature before. A simple request to the OPAC catalog for “Peak Shaving, Model Predictive Control” results into a lot of papers which are describing exactly this problem. So instead of describing the problem itself, it make sense to provide only the difference to the normal description given in the literature already. My recommendation is to add at least two literature references to the question. $\endgroup$ – Manuel Rodriguez Jun 12 at 20:25
  • $\begingroup$ Thanks for your comment. My question was more general, as it does not have to do specifically with the peak shaving problem, which can be framed in a large variety of ways. The core question is, rather: is it possible to frame the setting I described - where the large action space is the only thing that matters - as a RL problem? If so, how would it be possible to handle the absence of a state space? Would a standard policy gradient do the job? $\endgroup$ – FS93 Jun 13 at 0:09
  • $\begingroup$ What i want to explain is, that every problem was already described in the literature, and it's helps the reader to cite the sources. quote: “We visualize the Q function of applying RL without state-space augmentation and we find that it predictably struggles in the narrow passage.” Krishnan, Sanjay, et al. "Hirl: Hierarchical inverse reinforcement learning for long-horizon tasks with delayed rewards." arXiv preprint arXiv:1604.06508 (2016). $\endgroup$ – Manuel Rodriguez Jun 13 at 6:47

A stateless RL problem can be reduced to a Multiarmed Bandit (MAB) problem. In such a scenario, taking an action will not change the state of the agent.

So, this is the setting of a conventional MAB problem: at each time step, the agent selects an action to either perform an exploration or exploitation move. It then records the reward of the taken action and updates its estimation/expectation of the usefulness of the action. Then, repeats the procedure (selection, observing, updating).

To chose between exploration and exploitation moves, MAB agents adopt a strategy. The simplest one would probably be $\epsilon$-greedy which agent chooses the most rewarding actions most of the time (1-$\epsilon$ probability) or randomly selects an action ($\epsilon$ probability).

  • $\begingroup$ This makes sense. However, in the MAB scenario, you have a discrete action-space and the exploration/exploitation dynamics is much clearer and is normally approached (to my knowledge) with simple tabular methods. In my case, I have a very large matrix to be filled with real values, which would definitely require an approximation (a NN?) somewhere. Plus, considering the exploration dilemma: should I reshape the action-space to force the agent to change one entry of the matrix at the time (or one column at the time)? Otherwise, I do not see where and how exploration can happen in my case. $\endgroup$ – FS93 Jun 14 at 14:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.