What could be the cause of the drop in the reward in A3C?

Question

The mean episodic reward is generally increasing, but it has spontaneous drops, and I'm not sure of their cause.

The problem has a sparse reward, batch size=2000, entropy_coefficient=0.1, other hyper-parameters are pretty standard.

Has anyone seen this kind of behavior? What could be the cause these drops in the reward(not enough exploration, too sparse rewards, the state not expressive enough, etc.)?

Answer 1

I'll share my understanding so far. This kind of behavior is actually normal when using on-policy algorithms with a sparse final reward. Issue stems from the fact that once you get stuck in a behavior policy which does nothing (uses a "do nothing" action, for instance, until timeout), it's quite hard to get out of it, because you keep getting experiences that teach you nothing (no reward signal at all) and keeps you in the same policy. Possible mitigations:

• Encourage more exploration (in A3C, make the entropy loss coefficient bigger) in order to recover more quickly from this type of stationary behavior.
• Use an off-policy algorithm with a big enough replay buffer, so that even if you start behaving this way you still use experiences from a "healthy" older policy.

If you stick to a totally on-policy algorithm, making the batch_size bigger might help a little.