1
$\begingroup$

Twin Delayed DDPG (TD3) uses a double Q trick since the policy is deterministic like in DDPG, which is to mitigate the maximum overestimation bias in DDPG. However, in SAC, the policy is stochastic, even so, it still uses the double Q trick. So it confuses me that if it doesn't serve the purpose of mitigating the maximum overestimation bias, what is the real intention of this use?

$\endgroup$
5
  • $\begingroup$ Just because the policy is stochastic, it doesn’t mean that we aren’t trying to take a maximum over the Q function. $\endgroup$
    – David
    Aug 24, 2022 at 8:09
  • $\begingroup$ @DavidIreland, thanks for your reply! What you mean is that actually all algorithms have the issue of the maximum overestimation, just differ in degree? $\endgroup$ Aug 24, 2022 at 8:17
  • $\begingroup$ More or less. Algorithms like SAC and TD3 use the Q-function as the objective for the policy to maximise, so this problem is inherent even though it may not be explicitly obvious. SAC and TD3 use two critics to avoid overestimation bias. If you sample an action from your and the Q-function gives an overinflated estimate of the returns, then your policy will be updated too optimistically, and so using two Q-functions and taking the minimum over these two for the policy update is a way of trying to control the overestimation bias. $\endgroup$
    – David
    Aug 24, 2022 at 9:30
  • $\begingroup$ Of course, 2 is just chosen to trade-off between computational efficiency and dealing with the problem. You could choose any reasonable number (choosing a high number would give a pessimistic value, which would potentially slow down learning too, but I think has a less dramatic effect than overestimation bias). $\endgroup$
    – David
    Aug 24, 2022 at 9:31
  • $\begingroup$ @DavidIreland That means there are two sources of overestimation bias, one is from the Q function maximum (like in DDPG and TD3), the other is from the policy maximum? The latter seems like the main stream of such a bias in SAC, since SAC uses a stochastic policy, which chooses action randomly, rather than deterministically. If this holds, however, can I infer that the overestimation bias of policy maximum is not as severe as the Q function maximum? $\endgroup$ Aug 24, 2022 at 12:39

0

You must log in to answer this question.

Browse other questions tagged .