I studied the article "Demystifying Deep Reinforcement Learning" extensively during the last days, while trying to implement the proposed algorithms myself.
My goal is to have an agent learn by playing a simple board game against itself using the methods of deep reinforcement learning. The algorithm described in pseudo code in the chapter called "Deep Q-learning Algorithm" is straight forward, but I cannot wrap my head around the fact that the replay memory only gets initialized once and never gets cleared.
Besides the obvious issue of growing too large for the available memory, there seems to be a more fundamental flaw. In the beginning the games will appear almost random, because the Q-function just has been initialized randomly. This means that bad moves will hardly be punished by the opponent. It makes sense to learn from those random encounters in the beginning, as we do not have any better data to learn from and this actually describes the observed behavior of the environment very well.
But when the agent improves, the old memorized plays will no longer be valuable, as certain moves will no longer be played by the opponent and therefore the old games will no longer reflect the current behavior of the environment.
That's at least my interpretation. Now I am wondering if this is simply missing from the pseudo code in the article or if my way of thinking is wrong in this regard. My question is, if we need to flush the replay memory regularly in the given setup and how often it should be done?