Which Q function do we use to make our policy when using double Q learning?

Question

I know this might be arbitrary, but I couldn't find any good information on this. As we update 2 q function in double q learning I was curios whether we average, or sum them together to get our policy. Or whether we choose one over the other. It seems to me that when playing around with it typically one of the q functions would outperform the sum or average of both q functions.

Answer 1

For behaviour policy

It's kind of a free choice since the main rules for the behaviour policy are that

• It should cover all possible choices. This is handled outside of the Q function by adding some randomness

• For learning efficiency it should be close to the target policy. This is a loose, nice to have, requirement.

So any of the current Q being learned should do as the basis of $\epsilon$-greedy behaviour policy. Using average or alternating between them also works, but there is no requirement to match to the one being updated or the one being used to generate the maximising action for the update.

For final/output policy

The estimators are expected to converge to approximations of the true optimal action value function, with only minor differences, so you could use either at the end.

Using both and averaging them might offer a small boost to accuracy (similar to aggregating estimators in other scenarios).