I'm working on a project where I train a Q-learning agent to learn an optimal control policy for a water heater. I've set up a simulation which allows the agent to explore for one year. I then examine the results of the agent performance exploiting its optimal policy for the following year. The agent can perform the following actions (available actions depend on the state of the environment):
- Turn the electrical heating element on.
- Turn the electrical heating element off.
- Turn gas heating on.
- Turn gas heating off.
- Do nothing.
The goal of the agent reach the target temperature (50 deg C) when hot water is scheduled. The agent is rewarded for choosing actions which produce the lowest CO2 emissions (the CO2 emissions produced from electricity vary over time).
One of the issues I have noticed is that during the exploration phase, the agent tries a lot of weighted random actions which causes the water heater to overheat (>80 deg C). When the water heater overheats, it is not possible for the agent to perform further actions other than switching off heating and doing nothing. The agent is also punished for reaching the overheating tank state. The tank may remain in the overheated state for some time. It seems as if the tendency to overheat the tank during exploration is negatively impacting how the agent learns its policy as it reduces the number of experiences in other states.
Is there a term for this kind of situation during exploration in reinforcement learning? During exploration, the agent uses a chooses a softmax weighted random aciton. Are there alternative ways of choosing actions that may still allow for exploration while not reaching the overheating state?