Offline/Batch Reinforcement Learning: when to stop training and what agent to select

Question

Context:

My team and I are working on a RL problem for a specific application. We have data collected from user interactions (states, actions, rewards, etc.).

It is too costly for us to emulate agents. We decided therefore to concentrate on Offline RL techniques. For this, we are currently using the RL-Coach library by Intel, which offers support for Batch/Offline RL. More specifically, to evaluate policies in offline settings, we train a DDQN-BCQ model and evaluate the learned policies using Offline Policy Estimators (OPEs).

Problem:

In an Online RL setting, the decision of when to stop the training of an agent generally depends on the goal one wants to achieve (as described in this post: https://stats.stackexchange.com/questions/322933/q-learning-when-to-stop-training). If the goal is to train until convergence (of rewards) but no longer, then you could for example stop when the standard deviation of your rewards over the last n steps drops under some threshold. If the goal is to compare the performance of two algorithms, then you should simply compare the two using the same number of training steps.

However, in the Offline RL setting, I believe the conditions to stop training are not so clear. As stated above, no environement is directly available to evaluate our agents and the evaluation of the quality of the learned policy almost solely relies on OPEs, which are not always accurate.

For me, I believe that there are two different options that would make sense. I am unsure if both those options are actually equivalent though.

1. The first option would be to stop training when the Q-values have converged/reached a plateau (i.e. when the Q-value network loss has converged) -- if they ever do, as we don't really have any guarantee of this happening with artificial neural networks. If the Q-values do reach a plateau, this would mean that our agent has reached some local optimum (or in the best case, the global optimum).
2. The second option would be to only look at the OPEs reward estimation, and stop when they reach a plateau. However, different OPEs do not necessarily reach a plateau at the same time, as it can be seen in the figure below. In the Batch-RL tutorial of RL-Coach, it seems that they would simply select the agent at the epoch where the different OPEs give the highest policy value estimation, without checking that the loss of the network had converged or not (but this is only a tutorial, so I suppose we can't rely too much on it).

Questions:

• What would be the best criteria for choosing when to stop the training of an agent in an Offline-RL setting?
• Also, the performance of an agent often heavily depends on the seed used for training. To evaluate the general performance, I believe you have to run multiple training with different seeds? However, in the end, you still want only a single agent to deploy. Should you simply select the one having the highest OPEs values among all the runs?

P.S. I am not sure if this question should be splitted into two different posts, so please let me know if this is the case and I will edit the post!

Answer 1

We have deployed one project in the real world that uses offline RL algorithms. Evaluating the performance of a policy is indeed a very tricky problem. Unfortunately, most existing OPE method is not really matured enough for many practical problems, especially when evaluating relative complex tasks and policies. The final solution we use in the end is actually a combined approach:

• Train multiple policies with different seeds and initial hyperparameters. Compared with online RL algorithms, most existing offline RL policy learning, even the most performant one, such as CQL or Fisher-BRC have very large policy variance during training. The reasons could be due to not be able to generalize well at unseen data (still severe distributional shift during evaluation) as well as training instability. Training multiple policies is a must-do step.
• The policies should be trained until Q-values have converged/reached a plateau. Training offline RL with too many steps usually does not lead to good performance. Training until Q-value converges is the best practice one can make for most cases. Do not rely too much on OPE, as most existing methods are not performing well right now.
• For OPE, the only method that works for our project is actually the simple fitted Q evaluation (FQE), which produces relatively reliable policy evaluation. This method is not perfect, but is relatively stable and can help rule out policies with low and falsely learned Q-values. It is helpful to filter out some of the bad policies, but not guaranteed to find the best policy. For other OPE methods, importance sampling-based methods are completely unusable due to large variance; doubly robust methods still involves the importance weights hence still suffers the high variance and inaccuracy issue; marginalized importance sampling (MIS) based methods theoretically have lower variance (e.g. DualDice and other Dice family method), but in our experiments not very stable and also hard to train. The only method we find reasonable is FQE, but it can only help you filter out some of the truly bad policies.
• The final approach we use for policy selection is actually as follows. Train multiple policies using offline RL. Fit an ensemble dynamics/reward model using offline data, and rollout a few steps with the trained policies. If a policy leads to unreasonable states (based on domain knowledge) or a reward drop in most of the dynamics models, this policy is removed from the candidate set. Run FQE for all the policies in the candidate set, and filter out policies with very low Q values. The above two-step will help you remove 50-80% of not-so-well policies. The resulting policies are then deployed and test in a real-world environment. Although the above process cannot provide the best policy, it helps narrow down the set of policies that need to be tested in the real world (unfortunately, this is unavoidable given the current capability of offline RL algorithms and off-policy evaluation methods).
• The following paper actually conducted a series of empirical studies on multiple OPE methods, which might be helpful: "Voloshin C, Le HM, Jiang N, Yue Y. Empirical study of off-policy policy evaluation for reinforcement learning. 2019."