Can I train a DQN on the same dataset for multiple epochs?

Question

I am trying to learn about reinforcement learning and chose the stock market to experiment with. I have minute by minute historical data on a particular stock for the past 20 years. I am using a generator to feed the data into my DQN. I've been running some automated tuning on the hyperparameters and seem to have found some good values.

Now I am wondering if I should be training on the dataset more than once or whether that would cause the network to simply memorize past experiences and cause overfitting. Is there a standard practice when it comes to training on historical data in regards to the number of epochs?

Edit: I'm not nessesarily looking for an answer to how many epochs I should be using, rather I'd like to know if running over the same data more than once is okay with DQNs

Answer 1

I'm assuming according to your question that you have a fixed batch, or in other words, there's no possibility of further exploration in your settings. If this assumption is true, you have what's known as Batch/Offline Reinforcement Learning.

First, let's check some aspects about this: in Offline RL once that there's no possibility regarding further exploration, your dataset must contain a brunch of situations to leads your system to learn a robust policy. Imagine that you're working with robots, and training them using a fixed batch extracted using real-world interactions. If this batch was collected from a system that works pretty well, it might not contain samples about non-desirable situations. Once this system is deployed in the real world, facing a non-desirable situation, your system could not "know" which action should be taken to "escape" from these non-desirable states, once it never saw this during the learning phase. To summarizing, your dataset should be large and representative.

Now we assume that your dataset is large and representative. So, another problem arises: fixed batch shift the distribution of samples, creating a kind of bias that can be extrapolated by many epochs.

Finally, your dataset is composed of expert demonstrations? In other words, do you have the best actions on those samples in your dataset for each state?

The good thing about a fixed batch with real-world data is that it can booster the learning process once it contains the true dynamics of the real system. But be careful regarding the distribution of samples and training hyperparameter. Resuming, there's not a recipe for all problems, it depends on your case. I'll let two references that can be useful and where this answer is based:

References:

Nair, A., Dalal, M., Gupta, A., & Levine, S. (2020). Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359.

Fujimoto, Scott, David Meger, and Doina Precup. "Off-policy deep reinforcement learning without exploration." International Conference on Machine Learning. PMLR, 2019.