I am working on a deep reinforcement learning problem. The policy network has the same architecture as the one Deepmind published in 'Playing Atari with Deep Reinforcement Learning'. I am also using Prioritized Experience Replay. In the initial stage the behavior seems to be normal, i.e the agent is learning gradually. However, after a while the rewards suddenly go down by a lot. The TD erros also seem to be going up at the same time. I'm not sure how to interpret this problem.
My hypotheses are:
- The policy network is overfitting
- Some filters fail to activate thereby misrepresenting the state information
I would really appreciate if you guys could give me some tips to narrow down this problem debug it. Cheers.