1 vote
Accepted

Why do we need to have two heads in D3QN to obtain value and advantage separately, if V is the average of Q values?

Regarding your first question, $$V^{\pi}(s) = \sum_{a \in A}\pi(a|s)Q^{\pi}(s,a)$$ so recovering the value function from Q really depends on what policy $\pi$ you are using. Hence, you can't really ...
calveeen's user avatar
  • 1,261
1 vote

Questions on the identifiability issue and equations 8 and 9 in the D3QN paper

Yes, you're correct, if Equation 8 is used it will only be possible to get estimates $\leq 0$ out of the term $$\left( A(s, a; \theta, \alpha) - \max_{a' \in \vert \mathcal{A} \vert} A(s, a'; \theta, ...
Dennis Soemers's user avatar
  • 10.3k

Only top scored, non community-wiki answers of a minimum length are eligible