4
$\begingroup$

I try to apply RL for a control problem, and I intend to either use Deep Q-Learning or SARSA.

I have two heating storage systems with one heating device, and the RL agent is only allowed to heat up 1 for every time slot. How can I do that?

I have two continuous variables $x(t)$ and $y(t)$, where $x(t)$ quantifies the degree of maximum power for heating up storage 1 and $y(t)$ quantifies the degree of maximum power for heating up storage 2.

Now, IF $x(t) > 0$, THEN $y(t)$ has to be $0$, and vice versa with $x(t)$ and $y(t)$ element $0$ or $[0.25, 1]$. How can I tell this to the agent?

One way would be to adjust the actions after the RL agent has decided about that with a separate control algorithm that overrules the actions of the RL agent. I am wondering if and how this can be also done directly? I'll appreciate every comment.

Update: Of course I could do this with a reward function. But is there not a direct way of doing this? Because this is actually a so called hard constraint. The agent is not allowed to violate this at all as this is technically not feasible. So it will be better to tell the agent directly not to do this (if that is possible).

Reminder: Any further comments on that?

$\endgroup$
2
  • $\begingroup$ Hello @PeterBe, if I understand correctly, yes, an off-policy algorithm fit your problem. E.g. Q-Learning (but maybe a DDPG for the continuous case), SARSA is an off policy so you couldn't do that. Could you please tell me how you define your action space and state space? $\endgroup$
    – Pulse9
    yesterday
  • $\begingroup$ @Pulse9: Thanks for your comment. A simplified basic version of my problem is described here ai.stackexchange.com/questions/28888/… . You wrote "SARSA is an off policy so you couldn't do that"? Why can I not use an off policy learning algorithm for my problem? Or do you just want to say I can't use an off-policy algorithm to tell the RL agent some rules directly without any reward function (but I still can use an off-policy algorithm for my general problem)? $\endgroup$
    – PeterBe
    yesterday
3
$\begingroup$

You could just tweak your reward function to include this restrictions.

In the most simple case, you could reward your agent -1 if $x(t) > 0$ and $y(t) \neq 0$.

The scale of your negative reward depends on your general reward scaling of course.

$\endgroup$
3
  • $\begingroup$ Thanks tnfru for your answer. Of course I could do this with a reward function. But is there not direct way of doing this? Because this is actually a so called hard constraint. The agent is not allowed to violate this at all as this is technically not feasible. So it will be better to tell the agent directly not to do this (if that is possible) $\endgroup$
    – PeterBe
    Oct 15 at 12:27
  • $\begingroup$ Your question might be misleading then. without any constraints implies you don't want to do this as a hard constraint. I'd suggest then, that you overrule the agent and give negative reward so the agent stops selecting the action and it is never exectued. $\endgroup$
    – tnfru
    Oct 15 at 18:31
  • $\begingroup$ Thanks tnfru for your comment and effort. I really appreciate it. My question is if there is a direct way of telling this to the agent without any reward function. As far as I understand your answer you tend to say that there is not direct way of doing this? $\endgroup$
    – PeterBe
    Oct 18 at 12:07
0
$\begingroup$

I'm not an expert, but as far as I understand, you should use an off-policy algorithm, the difference between is:

On-Policy: The agent learns the value function according to the current action derived from current the policy being used. Off-Policy: The agent learns the value function according to the action derived from another policy.

This means that you can use another policy to explore. For example, if you use a Q-Learning (not your case because of the continuos values of your problem) that is an off-policy approach, you can explore with a particular policy to get the actions (you can only select valid actions) then you can update your q-table with the Q-Learning equation.

In your case you can use an off-policy deep approach. I suggest DDPG/TD3, you can look about some of them briefly here.

The idea is to use an exploration policy, the one you restrict to only select valid values (hard-constraint), and integrate the State, Action, Reward, State' in the replay buffer. The Stable_Baseline library doesnt allow that, but you could check the original source code of TD3.

Edit1:

If you see in the Q learning algorithm, the e-greedy consist on selecting with a probability of $\epsilon$ $a \gets \text{any action}$, and with $1-\epsilon$ the $a \gets max_{a}Q(s,a)$. This $\text{any action}$ is the part of the code that you use this "controller" to only select random (but valid) actions. This is because you want to explore but only explore with valid actions. Then the Q learning can "exploit" picking the best action from the exploration you did before. Now, for your case with continuos actions, you can use DDPG/TD3 to do something similar but you store these valid actions in a replay buffer, so your Neural Network can learn for this "data" of only valid actions.

Q learning

$\endgroup$
5
  • $\begingroup$ Thanks Pulse9 for your answer. I don't understand why I can't use Q-learning for that? I intend to use deep-Q-learning to map the inputs (states) to the outputs (actions). Further your wrote "The idea is to use an exploration policy, the one you restrict to only select valid values ". This is exactly my question. How can I restrict the agent to only select valid values? Of course I can just add another superior controller that can overrule the actions of the RL agent. But I would like to know if and how I can directly tell this to the RL agent while learning $\endgroup$
    – PeterBe
    12 hours ago
  • $\begingroup$ Moreover I have some problems understanding your explanations about on and off policy learning. You wrote "Off-Policy: The agent learns the value function according to the action derived from another policy." What do you mean by another policy in this context? How can I change the policy? $\endgroup$
    – PeterBe
    12 hours ago
  • $\begingroup$ I just improved my answer $\endgroup$
    – Pulse9
    12 hours ago
  • $\begingroup$ Thanks for your answer Pulse9. If I understood correctly your advice is to implement the Q-learning agent on my own and not use e.g. the DQNAgent from Keras? I would have to admit that I would not know how to implement this agent on my own and I assume that the effort would be way to high for that. Is it not possible to tell a predefined agent about some rules directly without implementing the whole agent from scratch? $\endgroup$
    – PeterBe
    12 hours ago
  • $\begingroup$ I forgot to specify the algorithms that I recommend for your case (DDPG/TD3) with continuos actions, the current libraries (Keras/Stable baseline...) do not allow you to define these exploration policies, but you can use others code from the authors that is ready to use, for example here: github.com/sfujim/TD3 $\endgroup$
    – Pulse9
    12 hours ago

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.