I've been looking for ways to improve my DQN. That is when I found the Double DQN algorithm. After looking at explanatory videos and posts, I've seen conflicting information:

  1. The Double DQN algorithm has two separate policies Q1 and Q2 with separate replay memories. They alternate in training and use the other network to get the Q values of the future state-action pair.

Q1(st, a) = r + γ * Q2(st+1, argmax Q1(st+1))

  1. The Double DQN algorithm works like a normal DQN with a policy and a target network (a copy of the policy lagging a few steps behind), and uses it to evaluate the Q value of the future state-action pair, chosen by the policy.

Q(st, a) = r + γ * Qtarget(st+1, argmax Q(st+1))

Which one of these solutions is correct? If both of them, which one is "better"? Or is there a third option I haven't considered?

Thanks in advance. Sorry for my math if the equations aren't correct. I hope they at least convey the intended meaning.

  • $\begingroup$ You can use mathjax on this site to format your formulas. $\endgroup$
    – nbro
    Commented Mar 21 at 14:44
  • $\begingroup$ For the record I did a short experiment on DQN with two policies. i.e. using 2 different policies with totally independent collection of history. My results were not statistically different than DDQN with a target network. Your results may vary. $\endgroup$
    – foreverska
    Commented Mar 21 at 14:57

1 Answer 1


The canonical DoubleDQN uses the target network. I've not seen the first version used anywhere in the deep RL literature, but it looks like what one would do if they were to take the original Double Q-Learning algorithm and place it exactly into the Deep RL format. The reason they don't do this in the Double DQN paper is because it would be very expensive to train and maintain two networks, when you are using a target network anyway. I believe they found that the target network worked well enough at reducing the maximisation bias without much more computational cost to the original DQN.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .