2
$\begingroup$

I am training a model using A2C with stable baselines 2. When I increased the timesteps I noticed that episode rewards seem to reset (see attached plot). I don´t understand where these sudden decays or resets could come from and I am looking for practical experience or pointers to theory what these resets could imply.

Plot of eposide rewards

$\endgroup$
1
  • $\begingroup$ could you please add some more information, e.g. what environment are you using, what is the reward space for that env, what exactly is the plot showing? $\endgroup$
    – pi-tau
    Aug 14, 2023 at 9:31

2 Answers 2

0
$\begingroup$

i've seen this in learning using deep Q network. some tips may help you came over this problem :

  • use some remembering mechanisms like replay buffer. some times agent forget what it has been learnt. reply buffer remembers the agent what he saw at several episodes ago.
  • something else that worked for me was changing the optimizer. As DQN article says , using RMSprop is very useful for learning neural network based agents.

if these tips dont help you, please give more information about your agent.

$\endgroup$
3
  • $\begingroup$ thank you. Indeed trying DQN instead of A2C seems to help. If I use a large replay buffer with DQN I don't get the resets, while with a smaller replay buffer I still get resets. Could this effect also be related to the very large variance I seem to get in episode rewards? (the individual points in the plot have a wide distribution along they y axis) $\endgroup$
    – qwertz
    Feb 25, 2022 at 20:34
  • $\begingroup$ this question needs more investigate on the network. But maybe it's right because input-output normalization of a network may affect the network. ‌Besides changing the optimizer may help you too. I suggest test RMSprop too. My intuition says me that RMSprop have too flexibility at the begging and then becomes more stable at the end of learning. this helps RL agent to not forget the first experiences. $\endgroup$ Feb 26, 2022 at 8:14
  • $\begingroup$ A2C is on-policy, so cannot use a long term replay table. It is common practice to run multiple environments at once instead, and wait until there's enough variety to calculate policy and value updates without too much correlation $\endgroup$ Aug 15, 2023 at 15:59
0
$\begingroup$

I ran into this and learned that not only is observation normalization is as important as reward normalization.

The y-axis is showing this env reward is much higher than 1.0

Try rescaling your env reward such that it always falls within the range -1 to 1.

The reason why reward normalization is required is it affects the policy loss scaling. If you look at your policy loss you will also notice it much higher than 1.0. A policy loss that high makes training the neural network unstable

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .