# How to Weigt Constraints in A Control Problem with Reinforcement Learning

I have a control problem for a heating device of a building with the goal to minimize the electricity costs for one day under a varying price for electricity in every hour. (more details can be seen here as well: Reinforcement learning applicable to a scheduling problem?).

I also want to test two further goals (minimize peak load and maximize PV self-consumption rate).

My problem also has about 10 constraints that should not be violated. I have two main questions about how to integrate the constraints into the Reinforcement Learning agent:

Here are my two main questions (with following minor questions):

(1) Basically I have three goals with normalized rewards between 0 and 1 for every time-slot and I have 10 constraints.

Should the constraints reward also be normalized for all 10 constraints? And then should I choose a higher weight for the most important constraint than for all three goals combined such that a constraint violation is more crucial than getting a better objective value for all the three goals?

(2) Is it also possible to tell the Reinforcement Learning agent some rules directly without any constraints?

E.g. I have two storage systems, and the agent is only allowed to heat up 1 for every time-slot. Further, the agent should not start and stop heating up frequently (like around four starts of the device daily is desirable).

Can I explicitly tell these rules to the agent? Or do I have to do it indirectly by calculating a reward for every of these constraints and incorporate the weighted reward into the overall reward function of the agent?

I'll appreciate any suggestion and comment.

Should the constraints reward also be normalized for all 10 constraints?

You should choose a "natural" balance between rewards where possible.

If you have many separate goals to take account of, ideally you should convert them all into some comparable metric that is meangful to the success of the agent. Such as a financial gain/loss, or energy gain/loss, or similar. You can normalise them after this, but the ratios between the values should be kept the same.

This is not always possible with constraints.

For strict constraints, you should ideally ensure that breaking any constraint will score worse than not breaking the constraint but scoring very badly at everything else. If your system is gaining positive rewards from operating within bounds (and it seems that it is from the description), then one simple way to achieve this is to terminate the episode early and score $$0$$ for breaking the constraint. If the constraint relates to an ongoing state measurement this may be the best option. That is because the agent learning how to "escape" from an unreachable state may not be useful.

For soft constraints, you need to decide a relative cost. For example your constraint:

the agent should not start and stop heating up frequently (like around four starts of the device daily is desirable)

Looks very much like a soft constraint - using words like "around" and "desirable". For something like that, I would probably allow four starts, then add largish penalty for each start beyond that. What that value is, should be related back to the natural balance between rewards and why you want this constraint.

As an aside, in order for the agent to learn about this constraint, you must add the number of starts so far for each device to the state. This is true for all constraints - there must be data inside the current state that the agent could use to predict that the constraint will come into play. It doesn't need to know the limit you are applying, but does need to know the current value of any variables used to decide whether a limit should be enforced.

I think you will also want to store what the last action was, or which device is currently on, so that the agent knows to keep it on in order not to waste one of its four uses of the device per day.

Is it also possible to tell the Reinforcement Learning agent some rules directly without any constraints?

Yes for absolute rules that should prevent the agent taking a specific action in the first place, and that will also be enforced in any production system. For example this rule:

I have two storage systems, and the agent is only allowed to heat up 1 for every time-slot.

Can be easily expressed by having three actions:

• $$a_0$$ no heating
• $$a_1$$ heat device 1
• $$a_2$$ heat device 2

Although you may want:

• $$a_0$$ do nothing
• $$a_1$$ start device 1 (and stop device 2 if running)
• $$a_2$$ start device 2 (and stop device 1 if running)
• $$a_3$$ stop whichever device is running

This second list may help the agent from flip/flopping devices during early stages of learning, and speed things up a little.

Other action sets could work too. The important thing is that there is nothing to gain by presenting the agent with action choices that you are ruling out as not possible immediately. It would mean adding more negative rewards and constraints to what already looks like a complex problem.

• Thanks a lot Neil for your answer. I have 3 questions on that. 1) About Hard constraints you wrote "terminate the episode early and score 0 for breaking the constraint." --> What do you mean by terminating the episode? To finish the RL and give a reward of 0? I think in my case this would happen many times such that the learning might suffer from that. 2) About the hard constraints you wrote "That is because the agent learning how to "escape" from an unreachable state may not be useful." --> What do you mean by that? Oct 11 '21 at 15:50
• 3) About the absolute rules I have not 3 but 2 actions (one for each story). Can I just directly change the actions in each iteration of the learning such that this constraint is not violated? For example if the RL agent wanted to assign a positive value for each 2 actions (which is not possible) I would just overwrite these proposed actions from RL with my own rule? Where would I have to do this? In the step function of the environment or shall I rather define a new separate control class that can overrule the actions of the RL agent. Oct 11 '21 at 15:50
• @PeterBe: For (1) Yes that's what I mean. For (2) I mean that if a state is not reachable, learning a policy decision for it may be a waste of time. For (3) I don't understand your comment, but I think you are saying that you have a continuous action vector, or at the least multiple settings for each heater. In either case this is not two actions, but more - potentially a lot more. However, you can still expand on the idea in the answer - the point is not to adjust things afterwards, but to construct the action space so that only valid actions are possible. Oct 11 '21 at 20:13
• Thanks Neil for your answer and effort. I really appreciate it. Regarding your answer to 2) " I mean that if a state is not reachable, learning a policy decision for it may be a waste of time." --> how can I make sure that this is not happening? Regarding your answer to 3) "the point is not to adjust things afterwards, but to construct the action space so that only valid actions are possible." --> how can I do that? I have two continuous variables x(t) and y(t). Now IF x(t) is >0 THEN y(t) has to be 0 and vice versa with x(t) and y(t) element 0 or [0.25,1]. How can tell this to the agent? Oct 12 '21 at 13:14
• For (2) the assumption is that you have a state space constraint. So for certain don't train on data that starts from unreachable states according to your constraint. For (3) I think that would be a good separate question on the site. You can also choose to interpret the lower figure as zero regardless in terms of effect. I don't know whether that would be better than your options for remodelling the action space to force only valid actions. Oct 12 '21 at 16:57