2
$\begingroup$

The use of target network is to reduce the chance of value divergence which could happen with off-policy samples trained with semi-gradient objectives. In Deep Q network, semi-gradient TD is used and with experience replay the training could diverge.

Target network is a slow changing network designed to slowly track the main value network. In Mnih 2013, it was designed to match the main network every $N$ steps. There is another way which slowly updates the weight in the direction to match the main network every step. To someone, the latter is called Polyak updates.

I have done some very limited experiments and seen that with the same update rate, e.g. $N=10$, Polyak update would update with the rate of 0.1, I usually see Polyak updates to give smoother progress and converge faster. My experiments are by no means conclusive.

I would thence ask if it is known which one to perform better, converge faster or has smoother progress, in a wider range of tasks and settings?

$\endgroup$
1
$\begingroup$

Most of papers and my experience support hard update, once per N steps. N is usually very bug, range in between 10^4 to 10^6. DQN training is slow. But that depend on the problem. If you DQN converge with soft update with weight ~.1 (N~10) your problem could be very simple.

$\endgroup$
  • $\begingroup$ Would you mind put some more detailed research of yours? I would like to see something like this technique has been used in 2019 by the SOTA or so. Thanks. $\endgroup$ – Phizaz Mar 1 at 4:02
  • $\begingroup$ I was doing DQN for car driving simulation. The main points: even thogh DQN is off policy it's effectively act similar to on-policy due to overfitting on latest samples(latest measired in millions). That could be helped by increasing target net update intervaland to some degree size of replay buffer. Its very important to randomly remix samples from replay buffer then assembling training minibatch. Overfitting causing DQN drift, wich is main enemy $\endgroup$ – mirror2image Mar 2 at 6:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.