# Reinforcement Learning without state space

I want to use Reinforcement Learning to optimize the distribution of energy for a peak shaving problem given by a thermodynamical simulation. However, I am not sure how to proceed as the action space is the only thing that really matters, in this sense:

• The action space is a 288x66 matrix of real numbers between 0 and 1. The output of the simulation and therefore my reward depend solely on the distribution of this matrix.
• The state space is therefore absent, as the only thing that matters is the matrix on which I have total control. At this stage of the simulation, no other variables are taken into consideration.

I am not sure if this problem falls into the tabular RL or it requires approximation. In this case, I was thinking about using a Policy gradient algorithm for figuring out the best distribution of the 288x66 matrix; however, I do not know how to behave with the "absence" of the state space. Instead of a tuple I would just have , is this even a RL-approachable problem? If not, how can I reshape it to make it solvable with RL techniques?

• The question sounds, if the problem is completely new and was never described in the literature before. A simple request to the OPAC catalog for “Peak Shaving, Model Predictive Control” results into a lot of papers which are describing exactly this problem. So instead of describing the problem itself, it make sense to provide only the difference to the normal description given in the literature already. My recommendation is to add at least two literature references to the question. – Manuel Rodriguez Jun 12 at 20:25
• Thanks for your comment. My question was more general, as it does not have to do specifically with the peak shaving problem, which can be framed in a large variety of ways. The core question is, rather: is it possible to frame the setting I described - where the large action space is the only thing that matters - as a RL problem? If so, how would it be possible to handle the absence of a state space? Would a standard policy gradient do the job? – FS93 Jun 13 at 0:09
• What i want to explain is, that every problem was already described in the literature, and it's helps the reader to cite the sources. quote: “We visualize the Q function of applying RL without state-space augmentation and we find that it predictably struggles in the narrow passage.” Krishnan, Sanjay, et al. "Hirl: Hierarchical inverse reinforcement learning for long-horizon tasks with delayed rewards." arXiv preprint arXiv:1604.06508 (2016). – Manuel Rodriguez Jun 13 at 6:47

To chose between exploration and exploitation moves, MAB agents adopt a strategy. The simplest one would probably be $$\epsilon$$-greedy which agent chooses the most rewarding actions most of the time (1-$$\epsilon$$ probability) or randomly selects an action ($$\epsilon$$ probability).