2
$\begingroup$

Suppose that I have a DQN agent, which has two neural networks: one is the primary Q network and the other is the target Q network. In every update, the target Q network is updated with a soft update strategy:

$$Q_{target} = (1-\tau) \times Q_{target} + \tau \times Q_{prime}$$

I saved the primary Q network's weights every $n$ episodes (say $n=10$), but, unfortunately, I did not save the target Q network's weights.

Say that my training process is aborted for some reason, and now I would like to continue the training using the latest saved weights. I can load the primary Q network's weights, but what about the target Q network's weights? Should I also use the latest primary Q network's weights for the target Q network's weights, or should I use the primary Q network's weights from several episodes ago, or how should it be?

$\endgroup$
0
2
$\begingroup$

Let's add a step index to your expression

$$Q_{target}^{n} = (1-\tau)Q^{n-1}_{target} + \tau\, Q^{n-1}_{primary}$$

We can expand it one step further

$$Q_{target}^{n} = (1-\tau)^2Q^{n-2}_{target} + (1-\tau)\tau\, Q^{n-2}_{primary} + \tau\, Q^{n-1}_{primary}$$

And further

$$Q_{target}^{n} = (1-\tau)^3Q^{n-3}_{target} + (1-\tau)^2\tau\, Q^{n-3}_{primary} + (1-\tau)\tau\, Q^{n-2}_{primary} + \tau\, Q^{n-1}_{primary}$$

So, I guess, we can write a general formula for $m$ steps behind like:

$$Q_{target}^{n} = (1-\tau)^{n-m}Q^{n-m}_{target} + \tau\,\sum_{i=0}^{m-1} (1-\tau)^i Q^{n-i-1}_{primary} $$

For $n-m$ large enough $(1-\tau)^{n-m}$ should be close to 0 and you should be able to approximately reconstruct your $Q_{target}^n$ using only the history of $Q_{primary}$es

Edit: I've missed that you only have snapshots with some step between them. This is not ideal, but a possible way out would be to use, say, a linear interpolation between snapshot points.

$\endgroup$
6
  • $\begingroup$ Sorry, I think I misunderstood previously. Did you mean if we set $m=0$ and have $n$ large enough, we will have $(1-\tau)^{n-m} = 0$ ? And also, I think you miss the fact that I only have saved weights of episodes in multiple of $n$s (say $n=10$, $Q^{10}_{primary}, Q^{20}_{primary}, ...$)? Nevertheless, your answer has very useful hints. $\endgroup$ – Sanyou Apr 9 at 12:35
  • $\begingroup$ @Sanyou Yes, sorry - was sloppy. I meant $n-m$ should be large enough. Edited the answer. $\endgroup$ – Kostya Apr 9 at 12:47
  • $\begingroup$ I see, we still need $m$, but with $n-m$ large enough the target still approximately becomes 0. With interpolation between snapshots, I think it will results in good enough approximation. I will accept your answer! Thanks a lot, excellent observation! $\endgroup$ – Sanyou Apr 9 at 12:50
  • $\begingroup$ Worth noting that although this approximation is possible, simply setting target network weights equal to primary network weights should also be OK, if the goal is to have a functional agent and interruptions are not very frequent. If your goal is to have a repeatable experiement even with interruptions, the OP would need to store a whole lot more. $\endgroup$ – Neil Slater Apr 9 at 15:15
  • $\begingroup$ @NeilSlater I was worried that in case when the latest primary network really spiked away from the running average it might be a bad starting point for the target network. Averaging over previous snapshots should be a more robust solution. $\endgroup$ – Kostya Apr 9 at 15:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.