I have a certain scheduling problem and I would like to know in general whether I can use Reinforcement learning (and if so what kind of RL) to solve it. Basically my problem is a mixed-integer linear optimization problem. I have a building with an electric heating device that converts electricity into heat. So the action vector (decision variable) is $x(t)$ which quantifies the electrical power of the heating device. The device has to take one decision for every minute of the day (so in total there are $24$ hours $\times 60$ minutes $= 1440$ variables). Each of those variables is a continuous variable and can have any value between $0$ and $2000 W$.

The state space contains several continuous variables:

  • External varying electricity price per minute: Between $0$ Cents and $100$ Cents per kWh (amount of energy)
  • Internal temperature of the building: Basically between every possible value but there is a constraint to have the temperature between $20 °C$ and $22 °C$
  • Heat demand of the building: Any value between $0 W$ and $10.000 W$
  • Varying "efficiency" of the electrical heating device between $1$ and $4$ (depending on the external outside temperature)

The goal is to minimize the electricity costs (under a flexible electricity tariff) and to not violate the temperature constraint of the building. As stated before, this problem can be solved by mathematical optimization (mixed-integer linear program). But I would like to know if you can solve this also with reinforcement learning? As I am new to reinforcement learning I would not know how to do this. And I have some concerns about this.

Here I have a very large state space with continuous values. So I can't build a comprehensive $Q-$table as there are to many values. Further, I am not sure whether the problem is a dynamic programming problem (as most/all?) of the reinforcement problems. From an optimization point of view it is a mixed-integer linear problem.

Can anyone tell me if and how I could solve this by using RL? If it is possible I would like to know which type of RL method is suitable for this. Maybe Deep-Q-Learning but also some Monte-Carlo policy iteration or SARSA? Shall I use model-free or model-based RL for this?

Reminder: Does nobody know whether and how I can use reinforcement learning for this problem? I'd highly appreciate every comment.

Can nobody give me some more information on my issue? I'll highly appreciate every comment and would be quite thankful for more insights and your help.

  • $\begingroup$ Is the output you need an advanced and fixed plan of all decisions for the day, or can decisions be made at each minute dynamically in response to state values at each time $t$? $\endgroup$ Aug 4 at 8:36
  • 1
    $\begingroup$ Hi Neil, thanks for your comment. Basically both options would be possible. The output could be a fixed plan (schedule) of all decisions using some predictions of external variables (like the outside temperature). But the output can also be made dynamically at each timeslot t based on the current status. $\endgroup$
    – PeterBe
    Aug 4 at 10:06
  • $\begingroup$ @NeilSlater: Hi Neil, it is possible with RL to only generate the actions for 1 timeslot or is it also possible to generate a schedule (=action for multiple time slots in the future)? $\endgroup$
    – PeterBe
    Aug 5 at 9:33
  • 1
    $\begingroup$ It is possible for an action to be a schedule for multiple time steps, but it is more natural in RL to consider one time step at a time. $\endgroup$ Aug 5 at 9:38

Details matter, and it is possible that your problem is best solved using classic control (solving the state equations) or operations research style optimisation. However, RL is also a good choice because it can be made to learn a controller that is not brittle when things go wrong.

One thing you will have to accept with RL is that the constraints will be soft constraints, even if you penalise them heavily. That is, you can expect that the internal temperature could drift outside of bounds. It definitely will during learning. A major design concern when framing the problem for reinforcement learning is how to weight the different rewards that represent your goals. You can weight your strict constraints higher, but at least initially they need to be low enough that the cost saving is not completely swamped.

I would suggest that your worst constraint failure penalty is slightly larger than the highest possible electricity cost for a single time step. That would mean the agent is always incentivised to spend money if it has to, as opposed to break the constraints, whilst still being able to explore what happens when it does break the constraints without having to cope with predicting large numbers.

There are lots of types of RL. Some are better at different kinds of problems. I would characterise your problem as you have described it as:

  • Episodic - but only for convenience of describing the problem. In fact your agent with a 24 hour episode will be incentivised to allow internal temperature to drop at the end of the 24 hours to save money, when it does not care what might happen immediately afterwards. Depending on price of electricity at that point, it could well be more optimal to spend more. This may only be a small difference from truly optimal behaviour, but you might play to strong point of RL by re-framing the problem as a continuing one (where mixed-integer linear optimisation may be harder to frame).

  • Continuous state space, with low dimensionality.

    • If prices are known in advance, you may want to augment the state space so that the agent knows how long it has at current price and whether the next price will be higher or lower. Alternatively, if they always follow the same time schedule, you could add the current time as a state variable. Either way, that allows the agent to take advantage of the temperature bounds. For instance, it could load up on cheap heating before a price hike, or allow the temperature to drop to minimum acceptable if cheaper electricity is about to become available.
  • Large, possibly continuous action space. You might want to consider approximating this to e.g. 21 actions (0 W, 100 W . . . 2000 W) as optimising this simpler variant will be easier to code (a DQN could do it), whilst it may not significantly affect optimality of any solution.

I don't think you could simplify your state space in order to use Q tables. So the DQN agent is probably the simplest that you could use here, provided you are willing to discretise the action space.

If you don't want to discretise the action space, then you will want to use some form of policy gradient approach. This will include a policy network that takes current state as input and then output a distribution of power level choices - e.g. a mean and standard deviation for the power choice. In production use you may be able to set the standard deviation to zero and use the mean as the action choice. A method like A3C can be used to do train such an agent.

I suggest that you discretise the actions and use a DQN-based agent to learn an approximate optimal policy for your environment. If that returns a promising result, you could either stop there or try to refine it further using continuous action space and A3C.

Also, you will want to practice using DQN on a simpler problem before diving in to your main project. Two reasonable learning problems here might be Cartpole-v1 and LunarLander-v2 which also has a continuous actions variant. Learning enough about setting up relevant RL methods to solve these toy problems should put you on a good footing to handle your more complex problem.

Keras documentation includes an example DQN for Atari Breakout, that you may be able to use as the basis for building your own code.

  • 1
    $\begingroup$ Keras is fine to build a DQN with. I would still advise you to study the toy problems, as per the answer, and I have picked three that are quite similar to your problem - continuous state space and simulations of physical environments. $\endgroup$ Aug 5 at 9:35
  • 1
    $\begingroup$ @PeterBe: DQN also learns an implied policy $\pi(s) = \text{argmax}_a \hat{q}(s,a, \theta)$, so when you have learned an (approximate) optimal action value function, you also have your (approximate) optimal policy. The other algorithms you suggest won't help you. SARSA is an alternative to Q learning that may work, but I suggest you stick to DQN for discretised actions. There are extensions to DQN that may help, although difficult to say much in advance. Use A3C, PPO or DDPG if you want to explore continuous action space. $\endgroup$ Aug 5 at 11:41
  • 1
    $\begingroup$ @PeterBe I suggest you ask a separate quiestion about your last issue because it is quite different to your original problem and this comment thread has gone on far too long. $\endgroup$ Aug 6 at 6:33
  • 1
    $\begingroup$ @PeterBe 1)You can do either, you will find that you end up writing an environment very similar to Gym ones if you write your own, but it is not a problem. 2) Yes, you can have compound actions. For more/better details please ask a new question on the site. Comments are for clarifying details on the answer, and I cannot give proper details here $\endgroup$ Aug 19 at 13:04
  • 1
    $\begingroup$ @PeterBe Sorry this comment thread has gone on far too long. Please ask follow-ups as a new question. You may also benefit from another point of view than mine. You could link this question and answer from your new question, but do please make it as self-contained as you can. $\endgroup$ Aug 30 at 15:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.