# Why do we need to have two heads in D3QN to obtain value and advantage separately, if V is the average of Q values?

I have two questions on the Dueling DQN paper. First, I have an issue on understanding the identifiability that Dueling DQN paper mentions: Here is my question: If we have given Q-values $$Q(s, a; \theta)$$ for all actions, I assume we can get value for state $$s$$ by:

$$V(s) = \frac {1} {|Q|} \sum_{a \in \mathcal{Q}} Q(s, a; \theta)$$ and the advantage by: $$A(s,a) = Q(s, a; \theta) - V(s), ~~~ \forall ~a ~in ~\mathcal{A}(s)$$

in which $$\mathcal{A}(s)$$ is the action space for state $$s$$. If this is correct, why do we need to have two heads in the network to obtain value and advantage separately?

and then obtain Q-value using

$$Q(s, a; \theta, \alpha, \beta) = V(s; \theta, \beta) + \left( A(s, a; \theta, \alpha) - \max_{a' \in | \mathcal{A} |} A(s, a'; \theta, \alpha) \right). \tag{8}$$

or $$Q(s, a; \theta, \alpha, \beta) = V (s; \theta, \beta) + \left( A(s, a; \theta, \alpha) − \frac {1} {|A|} \sum_{a' \in \mathcal{A}} A(s, a'; \theta, \alpha) \right). \tag{9}$$

Am I missing something?

My second question is why Dueling DQN does not use the target network as it is used in the DQN paper?

– nbro
Jan 28 at 16:31

Regarding your first question, $$V^{\pi}(s) = \sum_{a \in A}\pi(a|s)Q^{\pi}(s,a)$$ so recovering the value function from Q really depends on what policy $$\pi$$ you are using. Hence, you can't really recover the value function $$V(s)$$ from the $$Q(s,a)$$ values without knowing your policy distribution for state $$s$$.
However, you can recover $$Q^{\pi}(s,a)$$ values if we know $$V^{\pi}(s)$$ and $$A^{\pi}(s,a)$$. This is because $$A^{\pi}(s,a) = V^{\pi}(s,a) - Q^{\pi}(s,a)$$
by definition of advantage. And this is why you need 2 heads to recover the $$Q$$ values from the Value and Advantage functions. In the original paper, the author's do not use this direct equation to recover $$Q^{\pi}(s,a)$$ values due to "identifability" issue and the fact that both $$V^{\pi}(s)$$ and $$Q^{\pi}(s,a)$$ are only estimates.