My team and I are working on a RL problem for a specific application. We have data collected from user interactions (states, actions, rewards, etc.).
It is too costly for us to emulate agents. We decided therefore to concentrate on Offline RL techniques. For this, we are currently using the RL-Coach library by Intel, which offers support for Batch/Offline RL. More specifically, to evaluate policies in offline settings, we train a DDQN-BCQ model and evaluate the learned policies using Offline Policy Estimators (OPEs).
In an Online RL setting, the decision of when to stop the training of an agent generally depends on the goal one wants to achieve (as described in this post: https://stats.stackexchange.com/questions/322933/q-learning-when-to-stop-training). If the goal is to train until convergence (of rewards) but no longer, then you could for example stop when the standard deviation of your rewards over the last n steps drops under some threshold. If the goal is to compare the performance of two algorithms, then you should simply compare the two using the same number of training steps.
However, in the Offline RL setting, I believe the conditions to stop training are not so clear. As stated above, no environement is directly available to evaluate our agents and the evaluation of the quality of the learned policy almost solely relies on OPEs, which are not always accurate.
For me, I believe that there are two different options that would make sense. I am unsure if both those options are actually equivalent though.
- The first option would be to stop training when the Q-values have converged/reached a plateau (i.e. when the Q-value network loss has converged) -- if they ever do, as we don't really have any guarantee of this happening with artificial neural networks. If the Q-values do reach a plateau, this would mean that our agent has reached some local optimum (or in the best case, the global optimum).
- The second option would be to only look at the OPEs reward estimation, and stop when they reach a plateau. However, different OPEs do not necessarily reach a plateau at the same time, as it can be seen in the figure below. In the Batch-RL tutorial of RL-Coach, it seems that they would simply select the agent at the epoch where the different OPEs give the highest policy value estimation, without checking that the loss of the network had converged or not (but this is only a tutorial, so I suppose we can't rely too much on it).
- What would be the best criteria for choosing when to stop the training of an agent in an Offline-RL setting?
- Also, the performance of an agent often heavily depends on the seed used for training. To evaluate the general performance, I believe you have to run multiple training with different seeds? However, in the end, you still want only a single agent to deploy. Should you simply select the one having the highest OPEs values among all the runs?
P.S. I am not sure if this question should be splitted into two different posts, so please let me know if this is the case and I will edit the post!