0
$\begingroup$

I have a scheduling problem that I am trying to solve with RL (if you are interested in more details you can read about it here Reinforcement learning applicable to a scheduling problem?).

I have created an own environment (OpenAI-Gym) and I have trained the model for one specific day of the simulation. So I have 288 timesteps for one day (1 for each 5 minutes) and the simulation last until the end of the day. So the agent needs to make 288 decisions when having 1 control variable.

Now my question is whether it is possible to successively train an RL agent on the same environment for different days? The environment and reward function will stay the same but the input data will change as every day has different input data (temperature, heat demand, electricity price etc.). So I would like to train an agent for one day and then tell the agent to train on another day but not forget everything it has learned during the training of the first day. Thus I can make sure that the agent is not overfitting to one special input data but also has the ability to generalize and thus be applicable for different days.

Do you know if and how I can do this?

Reminder: As I still don't know how to tackle this problem I would like to remind you on this question. Can someone give me some advice on how to do this (I already got an answer but there are many questions open to me and the author does not seem to reply any more)?

$\endgroup$

1 Answer 1

2
$\begingroup$

You can mitigate catastrophic forgetting by storing the trajectories generated by the actors during training in a replay buffer. Then, you sample trajectories from that replay buffer. This way, each mini-batch of experience will contain data from multiple days.

There are many strategies to do this sampling, but you can start with uniform sampling. From what you're describing, it doesn't seem that storage is going to be an issue (288 data points per day is small), so you can keep all trajectories. If you can't afford to store all trajectories, then you should also design a strategy to remove them from the replay buffer.

You can refer to this handy guide describing how to implement a replay buffer in TensorFlow.

$\endgroup$
13
  • $\begingroup$ Thanks Raphael for your answer. I have to admit that I have problems understanding it. You mentioned the catastrophic forgetting. Actually, I don't want to forget the information from the previous days. And what do you mean by sampling trajectories from a replay buffer? And just for the record: I am not using TensorFlow for reinforcement learning so the posted link is not suitable for me (altough I appreciate that you have posted it). $\endgroup$
    – PeterBe
    May 23 at 7:29
  • $\begingroup$ Thanks Rapahel for your answer. Any comments to my last comment? I'll highly appreciate every further comment from you. $\endgroup$
    – PeterBe
    May 30 at 8:26
  • $\begingroup$ Any further comments? I have problems understanding your answer. Would you mind elaborating a little bit more on it (see my first comment). $\endgroup$
    – PeterBe
    Jun 7 at 9:06
  • $\begingroup$ Apologies I was on holidays :) The method I'm describing is supposed to prevent catastrophic forgetting. The idea is to store the 288 decisions the agent has made on day 1 (with associated state and reward). That's a trajectory. Then the next day, you train on a mixture of the decisions made on this day and those of day 1. On day 3 you train on a mixture of decisions made on this day and those of day 1 and 2, and so on. Is that clearer? $\endgroup$ Jun 9 at 18:14
  • $\begingroup$ Thanks for your comment Raphael. Actually I still don't understand it. Shall I train an agent and then test it on day 1 and store the values for each timeslot? But how shall this work. After training the model, I can just apply it on other days. How can I tell the model to futher learn another way and still somehow consider the values from the training of day 1? $\endgroup$
    – PeterBe
    Jun 10 at 11:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .