Is it necessary to clear the replay memory regularly in a DQN when an agent plays against itself?

Question

I studied the article "Demystifying Deep Reinforcement Learning" extensively during the last days, while trying to implement the proposed algorithms myself.

My goal is to have an agent learn by playing a simple board game against itself using the methods of deep reinforcement learning. The algorithm described in pseudo code in the chapter called "Deep Q-learning Algorithm" is straight forward, but I cannot wrap my head around the fact that the replay memory only gets initialized once and never gets cleared.

Besides the obvious issue of growing too large for the available memory, there seems to be a more fundamental flaw. In the beginning the games will appear almost random, because the Q-function just has been initialized randomly. This means that bad moves will hardly be punished by the opponent. It makes sense to learn from those random encounters in the beginning, as we do not have any better data to learn from and this actually describes the observed behavior of the environment very well.

But when the agent improves, the old memorized plays will no longer be valuable, as certain moves will no longer be played by the opponent and therefore the old games will no longer reflect the current behavior of the environment.

That's at least my interpretation. Now I am wondering if this is simply missing from the pseudo code in the article or if my way of thinking is wrong in this regard. My question is, if we need to flush the replay memory regularly in the given setup and how often it should be done?

Answer 1

The purpose of the replay memory in DQN and similar architectures is to ensure that the gradients of the deep net are stable and doesn't diverge. Limiting what memory to keep and how far into the past it reaches is a problem that arises in in practical AI deployments. Throwing it away usually isn't the best decision because the agent's gradients may become unstable. Rather, keeping a window of the last T timesteps/gameplays can keep enough information while ensuring that the data is relevant as the agent improves. There are some additional methods which can improve the results as well:

• Only keeping relevant timesteps (relevant depends highly on the application)
• Removing timesteps from memory with probability relative to their age (the oldest has the highest probability to be removed)

It should be noted, that a larger replay memory is usually better than a smaller one to ensure stability but although memory is cheap it isn't infinite.