# How to recover the target Q network's weights solely from the snapshots of the primary Q network's weights in DQN?

Suppose that I have a DQN agent, which has two neural networks: one is the primary Q network and the other is the target Q network. In every update, the target Q network is updated with a soft update strategy:

$$Q_{target} = (1-\tau) \times Q_{target} + \tau \times Q_{prime}$$

I saved the primary Q network's weights every $$n$$ episodes (say $$n=10$$), but, unfortunately, I did not save the target Q network's weights.

Say that my training process is aborted for some reason, and now I would like to continue the training using the latest saved weights. I can load the primary Q network's weights, but what about the target Q network's weights? Should I also use the latest primary Q network's weights for the target Q network's weights, or should I use the primary Q network's weights from several episodes ago, or how should it be?

$$Q_{target}^{n} = (1-\tau)Q^{n-1}_{target} + \tau\, Q^{n-1}_{primary}$$

We can expand it one step further

$$Q_{target}^{n} = (1-\tau)^2Q^{n-2}_{target} + (1-\tau)\tau\, Q^{n-2}_{primary} + \tau\, Q^{n-1}_{primary}$$

And further

$$Q_{target}^{n} = (1-\tau)^3Q^{n-3}_{target} + (1-\tau)^2\tau\, Q^{n-3}_{primary} + (1-\tau)\tau\, Q^{n-2}_{primary} + \tau\, Q^{n-1}_{primary}$$

So, I guess, we can write a general formula for $$m$$ steps behind like:

$$Q_{target}^{n} = (1-\tau)^{n-m}Q^{n-m}_{target} + \tau\,\sum_{i=0}^{m-1} (1-\tau)^i Q^{n-i-1}_{primary}$$

For $$n-m$$ large enough $$(1-\tau)^{n-m}$$ should be close to 0 and you should be able to approximately reconstruct your $$Q_{target}^n$$ using only the history of $$Q_{primary}$$es

Edit: I've missed that you only have snapshots with some step between them. This is not ideal, but a possible way out would be to use, say, a linear interpolation between snapshot points.

• Sorry, I think I misunderstood previously. Did you mean if we set $m=0$ and have $n$ large enough, we will have $(1-\tau)^{n-m} = 0$ ? And also, I think you miss the fact that I only have saved weights of episodes in multiple of $n$s (say $n=10$, $Q^{10}_{primary}, Q^{20}_{primary}, ...$)? Nevertheless, your answer has very useful hints. Commented Apr 9, 2021 at 12:35
• @Sanyou Yes, sorry - was sloppy. I meant $n-m$ should be large enough. Edited the answer. Commented Apr 9, 2021 at 12:47
• I see, we still need $m$, but with $n-m$ large enough the target still approximately becomes 0. With interpolation between snapshots, I think it will results in good enough approximation. I will accept your answer! Thanks a lot, excellent observation! Commented Apr 9, 2021 at 12:50
• Worth noting that although this approximation is possible, simply setting target network weights equal to primary network weights should also be OK, if the goal is to have a functional agent and interruptions are not very frequent. If your goal is to have a repeatable experiement even with interruptions, the OP would need to store a whole lot more. Commented Apr 9, 2021 at 15:15
• @NeilSlater I was worried that in case when the latest primary network really spiked away from the running average it might be a bad starting point for the target network. Averaging over previous snapshots should be a more robust solution. Commented Apr 9, 2021 at 15:20