2
$\begingroup$

There is a lot of discussion about the overestimation bias for Deep Q Learning and similar off-policy action value estimation algorithms like DDPG. This is why methods like Double DQN and TD3 were created.

But what I don't understand is, is it not true that every temporal difference estimation has an overestimation bias? If so, how come no one cares about it? In all literature I read, the overestimation bias is synonymous with Q learning methods. I can't see why this isn't a problem for all methods (other than methods that use Monte Carlo returns like REINFORCE).

For example, if we did Advantage Actor Critic (A2C) with a 1-step bootstrap, the policy update pushes the policy network toward the distribution that maximizes the estimated advantage:

$\DeclareMathOperator{\E}{\mathbb{E}}$ $\pi(s_t) \leftarrow \mathrm{one\_hot}\left (\mathrm{argmax}_a \; \E_{s_{t + 1}}\left[R_t + \gamma V(s_{t + 1}, \phi) - V(s_t, \phi) \right ] \right )$

If $V(s_{t + 1}, \phi)$ is noisy and for some $s_{t + 1}$ the value is significantly overestimated, then the actions leading to this state have an overestimated advantage value, and the policy is optimized toward the actions with overestimated advantage. This error propagates to the target value of the learned $V$ function as new trajectories are sampled using a biased policy that favors actions with artificially high advantage values, potentially destabilizing everything.

This problem seems exactly identical to the overestimation problem for Deep Q networks and DDPG... am I mistaken somewhere?

Improve this question
$\endgroup$
2
  • $\begingroup$ consider that first of all, all temporal difference learning methods are biased in some way... Q learning has the problem of overestimation due to the nature of the TD operator that is involving the $max$... now, if you have a network and not a table, that will be even more enhanced, due to the fact that now states are not independently represented $\endgroup$
    – Alberto
    Commented Nov 29, 2023 at 23:07
  • $\begingroup$ Agree that all TD estimates are said to be biased in the literature, and that replacing tables with networks amplify bias. I originally believed that Q learning is special because its update rule has an explicit $\max$ operator, as you said. But the literature also agrees that DDPG has the same overestimation problem, even though DDPG has no explicit $\max$ operator in the update rule. They say the $\max$ effectively still happens since the policy is optimized toward the best action, but I can't see why the same logic doesn't apply to other RL algorithms. $\endgroup$
    – Jerry Ding
    Commented Nov 30, 2023 at 6:36

0

Reset to default

You must log in to answer this question.