I've noticed if you have a small negative reward (e.g.,
-0.01) per step for idling and a series of idle steps, an agent seems to learn to trick GAE by learning a very low critic value target (
-0.99988735 in this particular case) so when the GAE delta is calculated, the result is:
delta = -0.01 + 0.99 * 1 * -0.99988735 - -0.99988735 = -1.12649999e-06
, which results in GAE being next to 0.
Formula (pretty standard):
delta = np.array([reward + gamma * (1 - done) * next_predicted_value - predicted_value for reward, done, next_predicted_value, predicted_value in zip(rewards, dones, next_values, critic_predictions)], dtype=np.float32)
Have you observed this kind of behavior? Any ideas on how to deal with it?