How and when should we update the Q-target in deep Q-learning?

Question

I have recently watched David silver's course, and started implementing the deep Q-learning algorithm.

I thought I should make a switch between the Q-target and Q-current directly (meaning, every parameter of Q-current goes to Q-target), but I found a repository on GitHub where that guy updates Q-target as follows:

$$Q_{\text{target}} = \tau * Q_{\text{current}} + (1 - \tau)*Q_{\text{target}}$$.

where $\tau$ is some number probably between 0 and 1.

Is that update correct or I miss something?

I thought after some iterations (e.g. 2000 iteration), we should update Q-target as: $Q_{\text{target}}=Q_{\text{current}}$.

Answer 1

The update form $\theta^{\prime} \leftarrow \tau \theta+(1-\tau) \theta^{\prime}$ (where $\theta'$ and $\theta$ represent the weights of the target network and the current network, respectively) does exist and is correct.

It is called soft update and it has been used in the Deep Deterministic Policy Gradient (DDPG) paper, which uses the concept of a target network like DQN. The authors of the paper state that:

The weights of these target networks are then updated by having them slowly track the learned networks: $\theta ' \leftarrow \tau \theta + (1 − \tau )\theta'$ with $\tau << 1$. This means that the target values are constrained to change slowly, greatly improving the stability of learning.

This update will be made in each time step as follows. For example, for $\tau= 0.001$, the new weights for the target network will take $0.1\%$ of the main network’s weights and $99.9 \%$ of the old target network weights. This does not go against the purpose of fixed target networks (which have been introduced to address the problem of “moving targets”). In fact, by keeping $99.9\%$ of the old target network weights, they can still be considered as fixed.

Seeing that this resulted in improvements in DDPG, some DQN implementations/tutorials started to use soft updates for the target network.

This is opposed to the hard update scheme used in the original DQN paper, i.e. the weights are copied every $C$ steps. This means that the target network is kept fixed for $C$ steps (10000 in the paper) and then gets a significant update.