# Question on identifiability in the “Dueling Network Architectures for Deep Reinforcement Learning” paper

I have two questions on the Dueling DQN paper. First, I have an issue on understanding the identifiability that Dueling DQN paper mentions:

Here is my question: If we have given Q-values $$Q(s, a; \theta)$$ for all actions, I assume we can get value for state $$s$$ by:

$$V(s) = \frac {1} {|Q|} \sum_{a \in \mathcal{Q}} Q(s, a; \theta)$$ and the advantage by: $$A(s,a) = Q(s, a; \theta) - V(s), ~~~ \forall ~a ~in ~\mathcal{A}(s)$$

in which $$\mathcal{A}(s)$$ is the action space for state $$s$$. If this is correct, why do we need to have two heads in the network to obtain value and advantage separately?

and then obtain Q-value using

$$Q(s, a; \theta, \alpha, \beta) = V(s; \theta, \beta) + \left( A(s, a; \theta, \alpha) - \max_{a' \in | \mathcal{A} |} A(s, a'; \theta, \alpha) \right). \tag{8}$$

or $$Q(s, a; \theta, \alpha, \beta) = V (s; \theta, \beta) + \left( A(s, a; \theta, \alpha) − \frac {1} {|A|} \sum_{a' \in \mathcal{A}} A(s, a'; \theta, \alpha) \right). \tag{9}$$

Am I missing something?

My second question is why Dueling DQN does not use the target network as it is used in the DQN paper?

Regarding your first question, $$V^{\pi}(s) = \sum_{a \in A}\pi(a|s)Q^{\pi}(s,a)$$ so recovering the value function from Q really depends on what policy $$\pi$$ you are using. Hence, you can't really recover the value function $$V(s)$$ from the $$Q(s,a)$$ values without knowing your policy distribution for state $$s$$.

However, you can recover $$Q^{\pi}(s,a)$$ values if we know $$V^{\pi}(s)$$ and $$A^{\pi}(s,a)$$. This is because $$A^{\pi}(s,a) = V^{\pi}(s,a) - Q^{\pi}(s,a)$$

by definition of advantage. And this is why you need 2 heads to recover the $$Q$$ values from the Value and Advantage functions. In the original paper, the author's do not use this direct equation to recover $$Q^{\pi}(s,a)$$ values due to "identifability" issue and the fact that both $$V^{\pi}(s)$$ and $$Q^{\pi}(s,a)$$ are only estimates.

Regarding your second question, I believe the author's appllied the Duelling architecture on Double Deep Q Networks, which is an improvement over the single DQN used by Minh et al in learning atari. I do think that you can still use a a target network as in the single DQN case if you wanted to.

• Thanks for the answer. About the target network, the double DQN paper uses a target network the same as DQN does (see the original paper of Double DQN or Appendix A of the Dueling DQN paper). But, dueling DQN uses a SARSA update to get the target value. They have mentioned that in Section 4.1. Still, not sure why they have not used target network in Dueling DQN. – Afshin Oroojlooy Mar 23 at 13:12
• I see what you mean. I am not very sure why they did not use a target network. It could be because that they are using the SARSA algorithm rather than Q-learning one. I think you can search for Deep SARSA methods to see whether the choice of network is similar. – calveeen Mar 23 at 14:31
• @AfshinOroojlooy The network used for section 4.1 is different than what they have used for the rest of the paper. If you notice, they have mentioned that in the paper. The reason for section 4.1 is due to illustrating the speed of training. At least as far as I know. – Media Mar 26 at 15:04
• @Media, Thanks for the reply. I noticed that they have used the DDQN framework for the Atari experiments, but was not sure which network is their proposal. – Afshin Oroojlooy Mar 26 at 15:15