I have two questions on the Dueling DQN paper. First, I have an issue on understanding the identifiability that Dueling DQN paper mentions:

enter image description here

Here is my question: If we have given Q-values $Q(s, a; \theta)$ for all actions, I assume we can get value for state $s$ by:

$$V(s) = \frac {1} {|Q|} \sum_{a \in \mathcal{Q}} Q(s, a; \theta)$$ and the advantage by: $$A(s,a) = Q(s, a; \theta) - V(s), ~~~ \forall ~a ~in ~\mathcal{A}(s)$$

in which $\mathcal{A}(s)$ is the action space for state $s$. If this is correct, why do we need to have two heads in the network to obtain value and advantage separately?

and then obtain Q-value using

$$Q(s, a; \theta, \alpha, \beta) = V(s; \theta, \beta) + \left( A(s, a; \theta, \alpha) - \max_{a' \in | \mathcal{A} |} A(s, a'; \theta, \alpha) \right). \tag{8}$$

or $$Q(s, a; \theta, \alpha, \beta) = V (s; \theta, \beta) + \left( A(s, a; \theta, \alpha) − \frac {1} {|A|} \sum_{a' \in \mathcal{A}} A(s, a'; \theta, \alpha) \right). \tag{9}$$

Am I missing something?

My second question is why Dueling DQN does not use the target network as it is used in the DQN paper?


Regarding your first question, $$V^{\pi}(s) = \sum_{a \in A}\pi(a|s)Q^{\pi}(s,a)$$ so recovering the value function from Q really depends on what policy $\pi$ you are using. Hence, you can't really recover the value function $V(s)$ from the $Q(s,a)$ values without knowing your policy distribution for state $s$.

However, you can recover $Q^{\pi}(s,a)$ values if we know $V^{\pi}(s)$ and $A^{\pi}(s,a)$. This is because $$A^{\pi}(s,a) = V^{\pi}(s,a) - Q^{\pi}(s,a)$$

by definition of advantage. And this is why you need 2 heads to recover the $Q$ values from the Value and Advantage functions. In the original paper, the author's do not use this direct equation to recover $Q^{\pi}(s,a)$ values due to "identifability" issue and the fact that both $V^{\pi}(s)$ and $Q^{\pi}(s,a)$ are only estimates.

Regarding your second question, I believe the author's appllied the Duelling architecture on Double Deep Q Networks, which is an improvement over the single DQN used by Minh et al in learning atari. I do think that you can still use a a target network as in the single DQN case if you wanted to.

  • 1
    $\begingroup$ Thanks for the answer. About the target network, the double DQN paper uses a target network the same as DQN does (see the original paper of Double DQN or Appendix A of the Dueling DQN paper). But, dueling DQN uses a SARSA update to get the target value. They have mentioned that in Section 4.1. Still, not sure why they have not used target network in Dueling DQN. $\endgroup$ Mar 23 at 13:12
  • 1
    $\begingroup$ I see what you mean. I am not very sure why they did not use a target network. It could be because that they are using the SARSA algorithm rather than Q-learning one. I think you can search for Deep SARSA methods to see whether the choice of network is similar. $\endgroup$
    – calveeen
    Mar 23 at 14:31
  • $\begingroup$ @Media, Thanks for the reply. I noticed that they have used the DDQN framework for the Atari experiments, but was not sure which network is their proposal. $\endgroup$ Mar 26 at 15:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.