2
$\begingroup$

I have recently watched David silver's course, and started implementing the deep Q-learning algorithm.

I thought I should make a switch between the Q-target and Q-current directly (meaning, every parameter of Q-current goes to Q-target), but I found a repository on GitHub where that guy updates Q-target as follows:

$$Q_{\text{target}} = \tau * Q_{\text{current}} + (1 - \tau)*Q_{\text{target}}$$.

where $\tau$ is some number probably between 0 and 1.

Is that update correct or I miss something?

I thought after some iterations (e.g. 2000 iteration), we should update Q-target as: $Q_{\text{target}}=Q_{\text{current}}$.

$\endgroup$
0

1 Answer 1

5
$\begingroup$

The update form $\theta^{\prime} \leftarrow \tau \theta+(1-\tau) \theta^{\prime}$ (where $\theta'$ and $\theta$ represent the weights of the target network and the current network, respectively) does exist and is correct.

It is called soft update and it has been used in the Deep Deterministic Policy Gradient (DDPG) paper, which uses the concept of a target network like DQN. The authors of the paper state that:

The weights of these target networks are then updated by having them slowly track the learned networks: $\theta ' \leftarrow \tau \theta + (1 − \tau )\theta'$ with $\tau << 1$. This means that the target values are constrained to change slowly, greatly improving the stability of learning.

This update will be made in each time step as follows. For example, for $\tau= 0.001$, the new weights for the target network will take $0.1\%$ of the main network’s weights and $99.9 \%$ of the old target network weights. This does not go against the purpose of fixed target networks (which have been introduced to address the problem of “moving targets”). In fact, by keeping $99.9\%$ of the old target network weights, they can still be considered as fixed.

Seeing that this resulted in improvements in DDPG, some DQN implementations/tutorials started to use soft updates for the target network.

This is opposed to the hard update scheme used in the original DQN paper, i.e. the weights are copied every $C$ steps. This means that the target network is kept fixed for $C$ steps (10000 in the paper) and then gets a significant update.

$\endgroup$
3
  • $\begingroup$ okay thank you so much.So in this repository this guy updates weights after every iteration, is that correct?It works on Lunalander perfectly, like you said he used soft update and changed target network after every iteration.Is that a coincidence that algorithm works or when you use soft update you can switch networks after every iteration? $\endgroup$ May 27, 2020 at 12:05
  • 1
    $\begingroup$ Yes, it is correct. Instead of waiting C steps to make an update, you update your target network in each step, as follows. For example, for $\tau= 0.001$, the new weights for the target network will take $0.1\%$ of the main network’s weights and $99.9 \%$ of the old target network weights. This is not a coincidence, because recall that fixed target networks have been introduced to address the problem of “moving targets”. So, by keeping $99.9\%$ of the old target network weights, they can still be considered as fixed. $\endgroup$ May 27, 2020 at 12:27
  • $\begingroup$ thanks a lot, i completely understand now. $\endgroup$ May 27, 2020 at 12:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .