Deep Reinforcement Learning: Rewards suddenly dip down

Question

I am working on a deep reinforcement learning problem. The policy network has the same architecture as the one Deepmind published in 'Playing Atari with Deep Reinforcement Learning'. I am also using Prioritized Experience Replay. In the initial stage the behavior seems to be normal, i.e the agent is learning gradually. However, after a while the rewards suddenly go down by a lot. The TD erros also seem to be going up at the same time. I'm not sure how to interpret this problem.

My hypotheses are:

1. The policy network is overfitting
2. Some filters fail to activate thereby misrepresenting the state information

I would really appreciate if you guys could give me some tips to narrow down this problem debug it. Cheers.

Answer 1

Without digging into your diagnostics more deeply, on it's face this seem like a local optima issue. Assuming you are optimizing via GD, there are many local optima that a network or agent in this case can converge to and stay at which will cause symptoms like seeing above.

With that being said, assuming this is our issue, here are some things you can try:

1. Regularization, try adding dropout or L2 and see how that affects convergence and learning.

2. Adjust network architecture, number of layers, nodes, etc.

3. Try a different type of RL(Q learning for example), this will be dependent on your problem of course.

4. Adjust starting seed. Assuming you have a static seed used for weight initialization, you will always converge to the same solution. It could be as simple as adjusting the seed value.

If all these steps fail, you likely have a deeper issue at work, and I would suggest coming back after with some additional detail if this does not succeed.