0
$\begingroup$

I've noticed if you have a small negative reward (e.g.,-0.01) per step for idling and a series of idle steps, an agent seems to learn to trick GAE by learning a very low critic value target (-0.99988735 in this particular case) so when the GAE delta is calculated, the result is:

delta = -0.01 + 0.99 * 1 * -0.99988735 - -0.99988735 = -1.12649999e-06

, which results in GAE being next to 0.

Formula (pretty standard):

delta = np.array([reward + gamma * (1 - done) * next_predicted_value - predicted_value for reward, done, next_predicted_value, predicted_value in zip(rewards, dones, next_values, critic_predictions)], dtype=np.float32)

Have you observed this kind of behavior? Any ideas on how to deal with it?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.