4
$\begingroup$

I am running a basic DQN (Deep Q-Network) on the Pong environment. Not a CNN, just a 3 layer linear neural net with ReLUs.

It seems to work for the most part, but at some point, my model suffers from catastrophic performance loss:

Catastrophic forgetting

  1. What is really the reason for that?

  2. What are the common ways to avoid this? Clipping the gradients? What else?

(Reloading from previous successful checkpoints feels more like a hack, rather than a proper solution to this issue.)

$\endgroup$
  • $\begingroup$ Try to increase size of replay buffer or increase time update interval for target network. $\endgroup$ – mirror2image Jun 1 at 6:56
  • $\begingroup$ hmmm, tried both. It still happens. I did analyse the replays though, and it looks like basically a lot of the early strong gains come from learning a single trick to win. And then later the net tries to win without using that trick (i guess it forgets about that at some point), leading to that strong underperformance, until it learns that trick (or another) again. Overall stability improves eventually. $\endgroup$ – Muppet Jun 1 at 20:48
  • 1
    $\begingroup$ There was good paper from deep mind about similar phenomena - contradicting objectives causing plateau in learning -"Ray Interference: a Source of Plateaus in Deep Reinforcement Learning" $\endgroup$ – mirror2image Jun 2 at 14:22
  • $\begingroup$ @mirror2image That sounds like it could be a good starting point for an answer. Please consider writing one based around a short summary of that paper. $\endgroup$ – John Doucette Nov 19 at 3:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.