# How to deal with GAE ineffectiveness because of critic value adaptation?

I've noticed if you have a small negative reward (e.g.,-0.01) per step for idling and a series of idle steps, an agent seems to learn to trick GAE by learning a very low critic value target (-0.99988735 in this particular case) so when the GAE delta is calculated, the result is:

delta = -0.01 + 0.99 * 1 * -0.99988735 - -0.99988735 = -1.12649999e-06


, which results in GAE being next to 0.

Formula (pretty standard):

delta = np.array([reward + gamma * (1 - done) * next_predicted_value - predicted_value for reward, done, next_predicted_value, predicted_value in zip(rewards, dones, next_values, critic_predictions)], dtype=np.float32)


Have you observed this kind of behavior? Any ideas on how to deal with it?