Is there an alternative to the use of target network?

Question

In the context of Deep Q Network, a target network is usually utilized. The target network is a slow changing network with a changing rate as its hyperparameter. This includes both replacement update every $N$ iterations and slowly update every iteration.

Since the rate is hard to fine tune manually, is there an alternative technique that can eliminate the use of target network or at least makes it less susceptible to the changing rate?

Answer 1

I have done some research and would like to share.

Generally to eliminate the use of target network one needs to show that training would be stable under off-policy semi-gradient.

There are two approaches that might work:

1. Experience reweighting
2. Constrained optimization

Experience reweighting

Probably the simplest idea is to use importance sampling ratio (Precup 2001) to multiply each sample. This would correct the off-policy sample distribution to be on-policy distribution. It has been shown (Sutton 2016) that on-policy samples lead to stability for semi-gradient. However, this line of work that corrects for sample distribution has high variance and does not work well in practice.

Another line of work aimed to partially correct the distribution only to the extent that would be provably stable is called Emphatic TD (Sutton 2016). The distribution is still mostly off-policy but is proved to be stable under linear function approximation.

Constrained optimization

The general wisdom comes from the fact that updating a value would also alter the target value since they share the same set of parameters. This problem is called over-generalization. To reduce this, Durugkar suggests that constraining the target Q-value after parameter update to be steady helps reduce divergence. Achiam 2019 provides a good overview of the problem, and suggests a way to make sure that the update is non-expansion (which should prevent divergence).

Sidenote

Some works have shown convergence under off-policy samples but only in tabular case. It takes much more to show that it is stable in function approximation (even a linear one).

Works like:

• Q($\lambda$) (Harutyunyan 2016), which augments the value function with a correction term
• Retrace($\lambda$) (Munos 2016), which extends the former by truncating the importance sampling ratio

are shown to work only in tabular case.

Answer 2

Never tried it, but there couple of approaches which may or may not help:

Distributional DQN ( not C51, other one, cant find ref right now typing from phone) with several output heads, chosen randomly

Multiple agents learning randomly on each other with some regularizer to pervent collapse to same net.

Both approach essentially try to"hide" or smear target.