5
$\begingroup$

I am running a basic DQN (Deep Q-Network) on the Pong environment. Not a CNN, just a 3 layer linear neural net with ReLUs.

It seems to work for the most part, but at some point, my model suffers from catastrophic performance loss:

Catastrophic forgetting

  1. What is really the reason for that?

  2. What are the common ways to avoid this? Clipping the gradients? What else?

(Reloading from previous successful checkpoints feels more like a hack, rather than a proper solution to this issue.)

Improve this question
$\endgroup$
4
  • $\begingroup$ Try to increase size of replay buffer or increase time update interval for target network. $\endgroup$
    – mirror2image
    Jun 1, 2019 at 6:56
  • $\begingroup$ hmmm, tried both. It still happens. I did analyse the replays though, and it looks like basically a lot of the early strong gains come from learning a single trick to win. And then later the net tries to win without using that trick (i guess it forgets about that at some point), leading to that strong underperformance, until it learns that trick (or another) again. Overall stability improves eventually. $\endgroup$
    – Muppet
    Jun 1, 2019 at 20:48
  • 1
    $\begingroup$ There was good paper from deep mind about similar phenomena - contradicting objectives causing plateau in learning -"Ray Interference: a Source of Plateaus in Deep Reinforcement Learning" $\endgroup$
    – mirror2image
    Jun 2, 2019 at 14:22
  • $\begingroup$ @mirror2image That sounds like it could be a good starting point for an answer. Please consider writing one based around a short summary of that paper. $\endgroup$
    – John Doucette
    Nov 19, 2019 at 3:03

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .