1
vote
Accepted
Why do we need to have two heads in D3QN to obtain value and advantage separately, if V is the average of Q values?
Regarding your first question, $$V^{\pi}(s) = \sum_{a \in A}\pi(a|s)Q^{\pi}(s,a)$$ so recovering the value function from Q really depends on what policy $\pi$ you are using. Hence, you can't really ...
- 1,291
1
vote
Questions on the identifiability issue and equations 8 and 9 in the D3QN paper
Yes, you're correct, if Equation 8 is used it will only be possible to get estimates $\leq 0$ out of the term
$$\left( A(s, a; \theta, \alpha) - \max_{a' \in \vert \mathcal{A} \vert} A(s, a'; \theta, ...
- 10.4k
Only top scored, non community-wiki answers of a minimum length are eligible