DQN with experience history to learn from already saved - which reward should I take?

Question

I want to train a DQN model in an off-policy fashion, where my behavior policy is an older agent. I have a big memory of a lot of episodes of this agent. Now I want to find a better policy using DQN. Now I am just wondering, in the "normal" DQN case you would use the experience replay buffer and would update behavior and target policy online (behavior not really online but with the time lag introduced after which these parameters are also updated).

In my case, I already have all the experience and would like to learn from it. Do you think it makes sense to use the exact same procedure in this context, so sampling one new action, state and immediate reward, follow up action or could it be better here to use the fact that all experience is already stored to exchange $R_{t+1} + \gamma \max Q(S_{t+1},a)$ with some more future information about the rewards (up to the point of Monte Carlo where you take $G_t$ so the discounted cumulative reward seen during the episode from point t onwards)?

Answer 1

More specifically than off-policy RL, you are looking at offline reinforcement learning techniques. In offline RL, all training data is known beforehand (stationary), which is in stark contrast to the usual non-stationarity of online RL data.

In my case, I already have all the experience

Are you absolutely sure of this? If your data doesn't contain transitions from high-reward regions of the state space, isn't generalizable to new situations, or is from very disparate and unconnected regions of the state space, then your agent could have trouble learning a good policy, let alone an optimal policy (see "When and why does offline RL work?" slide from this tutorial by Kumar and Levine).

If your data is indeed a good fit for offline RL, then there are certainly different updates that are recommended. In the same tutorial linked previously, the authors display some updates that are more conservative than traditional off-policy methods, especially when the data has high uncertainty. To answer your question - since you are currently using Q-learning, I suggest that you look into the conservative Q-learning (CQL) update as presented in the tutorial. To keep this answer brief and since I don't have much experience in this field, I will refer you to the corresponding survey paper for additional understanding and other possible updates.