You need to read this 2020 paper by Deepmind:
"Revisiting Fundamentals of Experience Replay"
Also, to add to the answer by @nbro
Assume you implement experience replay as a buffer where the newest memory is stored instead of the oldest. Then, if your buffer contains 100k entries, any memory will remain there for exactly 100k iterations.
Such a buffer is simply a way to "see" what was up to 100k iterations ago.
After the first 100k iterations you fill the buffer and begin "moving" it, much like a sliding window, by inserting new memories instead of the oldest.
The size of the buffer (relative to the total number of iterations you plan to ever train with) depends on "how much you believe your network architecture is susceptible to catastrophic forgetting".
A tiny buffer might force your network to only care about what it saw recently.
But an excessively large buffer might take a long time to "become refreshed" with good trajectories, when they finally start to be discovered. So the network would be like a university student whose book shelf is diluted with first-grade school books.
The student might have already decided that he/she wishes to become a programmer, so re-reading those primary school books has little benefit (time could have been spent more productively on programming literature) + it takes a long time to replace those with relevant university books.