1
$\begingroup$

I'm working on a project where I train a Q-learning agent to learn an optimal control policy for a water heater. I've set up a simulation which allows the agent to explore for one year. I then examine the results of the agent performance exploiting its optimal policy for the following year. The agent can perform the following actions (available actions depend on the state of the environment):

  • Turn the electrical heating element on.
  • Turn the electrical heating element off.
  • Turn gas heating on.
  • Turn gas heating off.
  • Do nothing.

The goal of the agent reach the target temperature (50 deg C) when hot water is scheduled. The agent is rewarded for choosing actions which produce the lowest CO2 emissions (the CO2 emissions produced from electricity vary over time).

One of the issues I have noticed is that during the exploration phase, the agent tries a lot of weighted random actions which causes the water heater to overheat (>80 deg C). When the water heater overheats, it is not possible for the agent to perform further actions other than switching off heating and doing nothing. The agent is also punished for reaching the overheating tank state. The tank may remain in the overheated state for some time. It seems as if the tendency to overheat the tank during exploration is negatively impacting how the agent learns its policy as it reduces the number of experiences in other states.

Is there a term for this kind of situation during exploration in reinforcement learning? During exploration, the agent uses a chooses a softmax weighted random aciton. Are there alternative ways of choosing actions that may still allow for exploration while not reaching the overheating state?

$\endgroup$
  • $\begingroup$ what is that you want the agent to learn? isn't switching off/doing nothing when the desired temperature is attained the end goal? $\endgroup$ – okkhoy May 10 '17 at 22:21
  • $\begingroup$ @okkhoy I want the agent to learn how to perform a set of actions which leads to the tank being at the target temperature when hot water is required. The problem is that during the exploration phase, the agent spends far more time in the negative overheating state. Once the water heater overheats, it can take some time for it to cool down to a point where the agent can perform other actions/begin to reach the target temperature. I've updated my post to clarify the goal of the agent. $\endgroup$ – Darkphenom May 10 '17 at 22:40
  • $\begingroup$ @Darkphenom Your project concept scenario does not require an agent,it's just require you to re-invent the wheel...in this case you don't require an agent..I did that project in c++[embedded systems programming] $\endgroup$ – quintumnia May 11 '17 at 10:19
  • $\begingroup$ @Darkphenom the way I see it, during the exploration phase, learning to do nothing when it has over heated is also a part of the learning process. if the tank got over heated, then the agent should know it should allow it to cool down. this way not doing anything is also useful, no? is there some detail missing which I am not grasping? $\endgroup$ – okkhoy May 11 '17 at 10:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.