# Does the DoubleDQN algorithm use a target network or two separate policies?

I've been looking for ways to improve my DQN. That is when I found the Double DQN algorithm. After looking at explanatory videos and posts, I've seen conflicting information:

1. The Double DQN algorithm has two separate policies Q1 and Q2 with separate replay memories. They alternate in training and use the other network to get the Q values of the future state-action pair.

Q1(st, a) = r + γ * Q2(st+1, argmax Q1(st+1))

1. The Double DQN algorithm works like a normal DQN with a policy and a target network (a copy of the policy lagging a few steps behind), and uses it to evaluate the Q value of the future state-action pair, chosen by the policy.

Q(st, a) = r + γ * Qtarget(st+1, argmax Q(st+1))

Which one of these solutions is correct? If both of them, which one is "better"? Or is there a third option I haven't considered?

Thanks in advance. Sorry for my math if the equations aren't correct. I hope they at least convey the intended meaning.

• You can use mathjax on this site to format your formulas.
– nbro
Commented Mar 21 at 14:44
• For the record I did a short experiment on DQN with two policies. i.e. using 2 different policies with totally independent collection of history. My results were not statistically different than DDQN with a target network. Your results may vary. Commented Mar 21 at 14:57