Does the DoubleDQN algorithm use a target network or two separate policies?

Question

I've been looking for ways to improve my DQN. That is when I found the Double DQN algorithm. After looking at explanatory videos and posts, I've seen conflicting information:

1. The Double DQN algorithm has two separate policies Q₁ and Q₂ with separate replay memories. They alternate in training and use the other network to get the Q values of the future state-action pair.

Q₁(s_t, a) = r + γ * Q₂(s_t+1, argmax Q₁(s_t+1))

2. The Double DQN algorithm works like a normal DQN with a policy and a target network (a copy of the policy lagging a few steps behind), and uses it to evaluate the Q value of the future state-action pair, chosen by the policy.

Q(s_t, a) = r + γ * Q_target(s_t+1, argmax Q(s_t+1))

Which one of these solutions is correct? If both of them, which one is "better"? Or is there a third option I haven't considered?

Thanks in advance. Sorry for my math if the equations aren't correct. I hope they at least convey the intended meaning.

Answer 1

The canonical DoubleDQN uses the target network. I've not seen the first version used anywhere in the deep RL literature, but it looks like what one would do if they were to take the original Double Q-Learning algorithm and place it exactly into the Deep RL format. The reason they don't do this in the Double DQN paper is because it would be very expensive to train and maintain two networks, when you are using a target network anyway. I believe they found that the target network worked well enough at reducing the maximisation bias without much more computational cost to the original DQN.