I've noticed if you have a small negative reward (e.g.,-0.01) per step for idling and a series of idle steps, an agent seems to learn to trick GAE by learning a very low critic value target (-0.99988735 in this particular case) so when the GAE delta is calculated, the result is:

delta = -0.01 + 0.99 * 1 * -0.99988735 - -0.99988735 = -1.12649999e-06

, which results in GAE being next to 0.

Formula (pretty standard):

delta = np.array([reward + gamma * (1 - done) * next_predicted_value - predicted_value for reward, done, next_predicted_value, predicted_value in zip(rewards, dones, next_values, critic_predictions)], dtype=np.float32)

Have you observed this kind of behavior? Any ideas on how to deal with it?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.