0
$\begingroup$

I am training a model using A2C with stable baselines 2. When I increased the timesteps I noticed that episode rewards seem to reset (see attached plot). I don´t understand where these sudden decays or resets could come from and I am looking for practical experience or pointers to theory what these resets could imply.

Plot of eposide rewards

$\endgroup$

1 Answer 1

0
$\begingroup$

i've seen this in learning using deep Q network. some tips may help you came over this problem :

  • use some remembering mechanisms like replay buffer. some times agent forget what it has been learnt. reply buffer remembers the agent what he saw at several episodes ago.
  • something else that worked for me was changing the optimizer. As DQN article says , using RMSprop is very useful for learning neural network based agents.

if these tips dont help you, please give more information about your agent.

$\endgroup$
2
  • $\begingroup$ thank you. Indeed trying DQN instead of A2C seems to help. If I use a large replay buffer with DQN I don't get the resets, while with a smaller replay buffer I still get resets. Could this effect also be related to the very large variance I seem to get in episode rewards? (the individual points in the plot have a wide distribution along they y axis) $\endgroup$
    – qwertz
    Feb 25 at 20:34
  • $\begingroup$ this question needs more investigate on the network. But maybe it's right because input-output normalization of a network may affect the network. ‌Besides changing the optimizer may help you too. I suggest test RMSprop too. My intuition says me that RMSprop have too flexibility at the begging and then becomes more stable at the end of learning. this helps RL agent to not forget the first experiences. $\endgroup$ Feb 26 at 8:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .