2
$\begingroup$

Suppose that I have a DQN agent, which has two neural networks: one is the primary Q network and the other is the target Q network. In every update, the target Q network is updated with a soft update strategy:

$$Q_{target} = (1-\tau) \times Q_{target} + \tau \times Q_{prime}$$

I saved the primary Q network's weights every $n$ episodes (say $n=10$), but, unfortunately, I did not save the target Q network's weights.

Say that my training process is aborted for some reason, and now I would like to continue the training using the latest saved weights. I can load the primary Q network's weights, but what about the target Q network's weights? Should I also use the latest primary Q network's weights for the target Q network's weights, or should I use the primary Q network's weights from several episodes ago, or how should it be?

$\endgroup$
0

1 Answer 1

2
$\begingroup$

Let's add a step index to your expression

$$Q_{target}^{n} = (1-\tau)Q^{n-1}_{target} + \tau\, Q^{n-1}_{primary}$$

We can expand it one step further

$$Q_{target}^{n} = (1-\tau)^2Q^{n-2}_{target} + (1-\tau)\tau\, Q^{n-2}_{primary} + \tau\, Q^{n-1}_{primary}$$

And further

$$Q_{target}^{n} = (1-\tau)^3Q^{n-3}_{target} + (1-\tau)^2\tau\, Q^{n-3}_{primary} + (1-\tau)\tau\, Q^{n-2}_{primary} + \tau\, Q^{n-1}_{primary}$$

So, I guess, we can write a general formula for $m$ steps behind like:

$$Q_{target}^{n} = (1-\tau)^{n-m}Q^{n-m}_{target} + \tau\,\sum_{i=0}^{m-1} (1-\tau)^i Q^{n-i-1}_{primary} $$

For $n-m$ large enough $(1-\tau)^{n-m}$ should be close to 0 and you should be able to approximately reconstruct your $Q_{target}^n$ using only the history of $Q_{primary}$es

Edit: I've missed that you only have snapshots with some step between them. This is not ideal, but a possible way out would be to use, say, a linear interpolation between snapshot points.

$\endgroup$
6
  • $\begingroup$ Sorry, I think I misunderstood previously. Did you mean if we set $m=0$ and have $n$ large enough, we will have $(1-\tau)^{n-m} = 0$ ? And also, I think you miss the fact that I only have saved weights of episodes in multiple of $n$s (say $n=10$, $Q^{10}_{primary}, Q^{20}_{primary}, ...$)? Nevertheless, your answer has very useful hints. $\endgroup$
    – Sanyou
    Apr 9, 2021 at 12:35
  • $\begingroup$ @Sanyou Yes, sorry - was sloppy. I meant $n-m$ should be large enough. Edited the answer. $\endgroup$
    – Kostya
    Apr 9, 2021 at 12:47
  • $\begingroup$ I see, we still need $m$, but with $n-m$ large enough the target still approximately becomes 0. With interpolation between snapshots, I think it will results in good enough approximation. I will accept your answer! Thanks a lot, excellent observation! $\endgroup$
    – Sanyou
    Apr 9, 2021 at 12:50
  • $\begingroup$ Worth noting that although this approximation is possible, simply setting target network weights equal to primary network weights should also be OK, if the goal is to have a functional agent and interruptions are not very frequent. If your goal is to have a repeatable experiement even with interruptions, the OP would need to store a whole lot more. $\endgroup$ Apr 9, 2021 at 15:15
  • $\begingroup$ @NeilSlater I was worried that in case when the latest primary network really spiked away from the running average it might be a bad starting point for the target network. Averaging over previous snapshots should be a more robust solution. $\endgroup$
    – Kostya
    Apr 9, 2021 at 15:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .