6
$\begingroup$

I try to apply RL for a control problem, and I intend to either use Deep Q-Learning or SARSA.

I have two heating storage systems with one heating device, and the RL agent is only allowed to heat up 1 for every time slot. How can I do that?

I have two continuous variables $x(t)$ and $y(t)$, where $x(t)$ quantifies the degree of maximum power for heating up storage 1 and $y(t)$ quantifies the degree of maximum power for heating up storage 2.

Now, IF $x(t) > 0$, THEN $y(t)$ has to be $0$, and vice versa with $x(t)$ and $y(t)$ element $0$ or $[0.25, 1]$. How can I tell this to the agent?

One way would be to adjust the actions after the RL agent has decided about that with a separate control algorithm that overrules the actions of the RL agent. I am wondering if and how this can be also done directly? I'll appreciate every comment.

Update: Of course I could do this with a reward function. But is there not a direct way of doing this? Because this is actually a so called hard constraint. The agent is not allowed to violate this at all as this is technically not feasible. So it will be better to tell the agent directly not to do this (if that is possible).

Reminder: Can anyone tell me more about this issue? I'll highly appreciate any further comment and will be quite thankful for your help. I will also award a bounty for a good answer.

$\endgroup$
2
  • $\begingroup$ Hello @PeterBe, if I understand correctly, yes, an off-policy algorithm fit your problem. E.g. Q-Learning (but maybe a DDPG for the continuous case), SARSA is an off policy so you couldn't do that. Could you please tell me how you define your action space and state space? $\endgroup$
    – Pulse9
    Oct 20 at 14:20
  • $\begingroup$ @Pulse9: Thanks for your comment. A simplified basic version of my problem is described here ai.stackexchange.com/questions/28888/… . You wrote "SARSA is an off policy so you couldn't do that"? Why can I not use an off policy learning algorithm for my problem? Or do you just want to say I can't use an off-policy algorithm to tell the RL agent some rules directly without any reward function (but I still can use an off-policy algorithm for my general problem)? $\endgroup$
    – PeterBe
    Oct 20 at 14:33
1
+50
$\begingroup$

I'm not an expert, but as far as I understand, you should use an off-policy algorithm, the difference between is:

On-Policy: The agent learns the value function according to the current action derived from current the policy being used. Off-Policy: The agent learns the value function according to the action derived from another policy.

This means that you can use another policy to explore. For example, if you use a Q-Learning (not your case because of the continuos values of your problem) that is an off-policy approach, you can explore with a particular policy to get the actions (you can only select valid actions) then you can update your q-table with the Q-Learning equation.

In your case you can use an off-policy deep approach. I suggest DDPG/TD3, you can look about some of them briefly here.

The idea is to use an exploration policy, the one you restrict to only select valid values (hard-constraint), and integrate the State, Action, Reward, State' in the replay buffer. The Stable_Baseline library doesnt allow that, but you could check the original source code of TD3.

Edit1:

If you see in the Q learning algorithm, the e-greedy consist on selecting with a probability of $\epsilon$ $a \gets \text{any action}$, and with $1-\epsilon$ the $a \gets max_{a}Q(s,a)$. This $\text{any action}$ is the part of the code that you use this "controller" to only select random (but valid) actions. This is because you want to explore but only explore with valid actions. Then the Q learning can "exploit" picking the best action from the exploration you did before. Now, for your case with continuos actions, you can use DDPG/TD3 to do something similar but you store these valid actions in a replay buffer, so your Neural Network can learn for this "data" of only valid actions.

Q learning

Edit 2:

In your custom environment you can define your action space like:

self.action_space = gym.spaces.Box(low=-1, high=1, shape=(1,))

Now, as you said, in the step function of your environment you can establish the x(t) and y(t)

maxX=10 #Depends on the maximum value of your x(t), I assigned a 10
maxY=10 #Depends on the maximum value of your y(t), I assigned a 10
x=0
y=0
if action>0:
    y=0
    x=action*maxX
elif action<0:
    x = 0
    # you need to multiply by -1 because your action is negative
    y = -1*action * maxY 
# do the rest of the code of your controler with x and y

In this way, your RL agent will learn which action (between -1 and 1) will get the best reward, but in the step function, you map the action [-1 +1] to your true values.

$\endgroup$
25
  • $\begingroup$ Thanks Pulse9 for your answer. I don't understand why I can't use Q-learning for that? I intend to use deep-Q-learning to map the inputs (states) to the outputs (actions). Further your wrote "The idea is to use an exploration policy, the one you restrict to only select valid values ". This is exactly my question. How can I restrict the agent to only select valid values? Of course I can just add another superior controller that can overrule the actions of the RL agent. But I would like to know if and how I can directly tell this to the RL agent while learning $\endgroup$
    – PeterBe
    Oct 21 at 12:41
  • $\begingroup$ Moreover I have some problems understanding your explanations about on and off policy learning. You wrote "Off-Policy: The agent learns the value function according to the action derived from another policy." What do you mean by another policy in this context? How can I change the policy? $\endgroup$
    – PeterBe
    Oct 21 at 12:48
  • $\begingroup$ I just improved my answer $\endgroup$
    – Pulse9
    Oct 21 at 13:11
  • $\begingroup$ Thanks for your answer Pulse9. If I understood correctly your advice is to implement the Q-learning agent on my own and not use e.g. the DQNAgent from Keras? I would have to admit that I would not know how to implement this agent on my own and I assume that the effort would be way to high for that. Is it not possible to tell a predefined agent about some rules directly without implementing the whole agent from scratch? $\endgroup$
    – PeterBe
    Oct 21 at 13:16
  • 1
    $\begingroup$ 1) If you can discretize your space, then is fine, you can use DQL or SARSA. 2) I was saying about the mapping so your agent can learn that positive values (0,1] represent an x>0 and y=0 and negative values [-1,0) represent a y>0 and x=0, but these actions are coded inside your environment, the agent will learn only de action a=[-1,1]. $\endgroup$
    – Pulse9
    Nov 14 at 20:56
4
$\begingroup$

You could just tweak your reward function to include this restrictions.

In the most simple case, you could reward your agent -1 if $x(t) > 0$ and $y(t) \neq 0$.

The scale of your negative reward depends on your general reward scaling of course.

$\endgroup$
4
  • $\begingroup$ Thanks tnfru for your answer. Of course I could do this with a reward function. But is there not direct way of doing this? Because this is actually a so called hard constraint. The agent is not allowed to violate this at all as this is technically not feasible. So it will be better to tell the agent directly not to do this (if that is possible) $\endgroup$
    – PeterBe
    Oct 15 at 12:27
  • $\begingroup$ Your question might be misleading then. without any constraints implies you don't want to do this as a hard constraint. I'd suggest then, that you overrule the agent and give negative reward so the agent stops selecting the action and it is never exectued. $\endgroup$
    – tnfru
    Oct 15 at 18:31
  • $\begingroup$ Thanks tnfru for your comment and effort. I really appreciate it. My question is if there is a direct way of telling this to the agent without any reward function. As far as I understand your answer you tend to say that there is not direct way of doing this? $\endgroup$
    – PeterBe
    Oct 18 at 12:07
  • $\begingroup$ To my knowledge none other than the overruling suggested above. $\endgroup$
    – tnfru
    Oct 26 at 17:31
0
$\begingroup$

When you take a step in the DQL process, you sample a move based on the estimated qualities of each possible action. During that step, you can restrict your sampling method to have probability 0 of choosing the forbidden action.

$\endgroup$
1
  • $\begingroup$ Thanks nnolte for your answer. I have to admit that I do not really understand how I can do what you suggest. Would you mind elaborating a little bit more on that? Here are some questions that I have: 1) How can I estimate the quality of each possible action and where shall I do this? In the step function (of Open AI Gym)? 2) How can I restrict the undesired actions to probability 0? 3) Is this approach not the same as having a super-ordinate additinal controller that can overrule the actions of the RL agent? $\endgroup$
    – PeterBe
    Nov 6 at 11:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.