# In DQN, updating target network every N steps or slowly update every step is better?

The use of target network is to reduce the chance of value divergence which could happen with off-policy samples trained with semi-gradient objectives. In Deep Q network, semi-gradient TD is used and with experience replay the training could diverge.

Target network is a slow changing network designed to slowly track the main value network. In Mnih 2013, it was designed to match the main network every $$N$$ steps. There is another way which slowly updates the weight in the direction to match the main network every step. To someone, the latter is called Polyak updates.

I have done some very limited experiments and seen that with the same update rate, e.g. $$N=10$$, Polyak update would update with the rate of 0.1, I usually see Polyak updates to give smoother progress and converge faster. My experiments are by no means conclusive.

I would thence ask if it is known which one to perform better, converge faster or has smoother progress, in a wider range of tasks and settings?