1
$\begingroup$

Target network in DQN is known to make the network more stable, and the loss is like "how good I'm now compared to using the target". What I don't understand is, if the target network is the stable one, why do we keep using/saving the first model as the predictor instead of the target?

I see in the code everywhere:

  • Model
  • Target model
  • Train model
  • Copy to target
  • Get loss between them

At the end, the model is saved and used for prediction and not the target.

$\endgroup$
1
$\begingroup$

The learning model and target model are only different by N steps (typically a few thousand) out the entire taining process. If the process is near complete, they will also be quite similar.

The target model is not inherently more stable in terms of producing "correct" or "better" Q values. Instead it is kept static for a period of time in order to stabilise the temporal difference (TD) target $R_{t+1} + \gamma \text{max}_{a'} \hat{q}(S_{t+1}, a', \theta^{-})$

Due to the copying stage that you listed:

  • If you return the target model, this is identical to returning the learning model from N steps beforehand.

  • Which in turn means that there was no point in doing the last N steps of training. You may as well have returned the learning model at the same point as copying.

Or another way of thinking about it: Returning the learning model at the end of the training process is identical to copying it to the target model as normal, then returning the target model immediately.

$\endgroup$
2
  • $\begingroup$ Maybe one practical suggestion would be to see if learning becomes "unstable" (in some sense) in the last $N$ steps. If that's the case, maybe you can just derive the policy from the target network. $\endgroup$ – nbro Nov 1 '20 at 11:55
  • $\begingroup$ @nbro: I'm not sure how that could be made practical. You can detect that changes are happening, but not why (it might be due to discovery of something real that needs adjusting for, or it might be catastrophic forgetting). The practical approach I take is to separately test candidate networks at snapshot points. I generally ignore the relationship with when the last copying was to the target network. However, due to taking snapshots at e.g. every 10,000 steps, and copying to target every 1,000 steps, often there is no difference between target and learning networks, and it's a non-issue. $\endgroup$ – Neil Slater Nov 1 '20 at 11:59
1
$\begingroup$

Target network is not more stable. Both networks are the same in the regard that no one is more stable than the other. The reason for using a target network is that your current network after each step is updated. So, by not using a target network and using just the current network, after each update the rewards for many states will be modified slightly. So, for a particular state, after each update, the reward will be modified which will lead to an unstable reward. This happens because when you update Q value function for state S, you may also slightly change the reward it predicts for state S' in the next step. So, each update to Q value function changes slightly the rewards for many other states, which makes the predicted rewards in all those states to be slightly unstable, as they are changed very often and you dont have a clear reward in those states.

However, by using a target network, your reward will be more stable(as you dont update your target network at each step, but you only update the current network).

So, by using a target network, the update is more stable, not the individual networks. And because the target network is an older version of the current network, it is sensible to use the current network as it was trained more than the target network

$\endgroup$
3
  • $\begingroup$ "This happens because when you update Q value function for state S, you may also slightly change the reward it predicts for state S'.", I don't see how this is true, if you are just updating Q(s, a). Maybe you mean that, if at the next iteration/time step, you use the just updated Q(s, a) to update Q(s', a), then you continue doing this, then you will somehow be oscillating. $\endgroup$ – nbro Nov 1 '20 at 11:53
  • $\begingroup$ that is true. By using the current network which was updated for a state to predict the reward for other states, the reward for those states will be different than the reward for those states before updating for a particular state. $\endgroup$ – Otniel Mercea Nov 1 '20 at 11:57
  • $\begingroup$ Yes, but that's different than what you wrote (in the sense that what you wrote is not fully clear), so I would edit your post to clarify that. I would also edit your post to clarify this sentence "Both networks are the same". Both networks have the same architecture, but they are not the same (i.e. do not have the same parameters' values), apart from when you just copied the main value network to the target network. $\endgroup$ – nbro Nov 1 '20 at 11:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.