I'm assuming according to your question that you have a fixed batch, or in other words, there's no possibility of further exploration in your settings. If this assumption is true, you have what's known as Batch/Offline Reinforcement Learning.
First, let's check some aspects about this: in Offline RL once that there's no possibility regarding further exploration, your dataset must contain a brunch of situations to leads your system to learn a robust policy. Imagine that you're working with robots, and training them using a fixed batch extracted using real-world interactions. If this batch was collected from a system that works pretty well, it might not contain samples about non-desirable situations. Once this system is deployed in the real world, facing a non-desirable situation, your system could not "know" which action should be taken to "escape" from these non-desirable states, once it never saw this during the learning phase. To summarizing, your dataset should be large and representative.
Now we assume that your dataset is large and representative. So, another problem arises: fixed batch shift the distribution of samples, creating a kind of bias that can be extrapolated by many epochs.
Finally, your dataset is composed of expert demonstrations? In other words, do you have the best actions on those samples in your dataset for each state?
The good thing about a fixed batch with real-world data is that it can booster the learning process once it contains the true dynamics of the real system. But be careful regarding the distribution of samples and training hyperparameter. Resuming, there's not a recipe for all problems, it depends on your case. I'll let two references that can be useful and where this answer is based:
Nair, A., Dalal, M., Gupta, A., & Levine, S. (2020). Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359.
Fujimoto, Scott, David Meger, and Doina Precup. "Off-policy deep reinforcement learning without exploration." International Conference on Machine Learning. PMLR, 2019.