In the paper describing TD3 (https://arxiv.org/abs/1802.09477), the authors say that they could not effectively address the Q-learning overestimation bias by using different networks for maximizing and estimating the next state Q value when estimating the target Q, even though the idea worked in the Double DQN paper.

They say the problem is due to the fact that the policy changes slowly when using an actor-critic architecture like DDPG.

But why would the quickness of the change in policy matter when it comes to the ability to reduce the overestimation bias?



You must log in to answer this question.