The mean episodic reward is generally increasing, but it has spontaneous drops, and I'm not sure of their cause.
The problem has a sparse reward, batch
entropy_coefficient=0.1, other hyper-parameters are pretty standard.
Has anyone seen this kind of behavior? What could be the cause these drops in the reward(not enough exploration, too sparse rewards, the state not expressive enough, etc.)?