I have done some research and would like to share.
Generally to eliminate the use of target network one needs to show that training would be stable under off-policy semi-gradient.
There are two approaches that might work:
- Experience reweighting
- Constrained optimization
Probably the simplest idea is to use importance sampling ratio (Precup 2001) to multiply each sample. This would correct the off-policy sample distribution to be on-policy distribution. It has been shown (Sutton 2016) that on-policy samples lead to stability for semi-gradient. However, this line of work that corrects for sample distribution has high variance and does not work well in practice.
Another line of work aimed to partially correct the distribution only to the extent that would be provably stable is called Emphatic TD (Sutton 2016). The distribution is still mostly off-policy but is proved to be stable under linear function approximation.
The general wisdom comes from the fact that updating a value would also alter the target value since they share the same set of parameters. This problem is called over-generalization. To reduce this, Durugkar suggests that constraining the target Q-value after parameter update to be steady helps reduce divergence. Achiam 2019 provides a good overview of the problem, and suggests a way to make sure that the update is non-expansion (which should prevent divergence).
Some works have shown convergence under off-policy samples but only in tabular case. It takes much more to show that it is stable in function approximation (even a linear one).
- Q($\lambda$) (Harutyunyan 2016), which augments the value function with a correction term
- Retrace($\lambda$) (Munos 2016), which extends the former by truncating the importance sampling ratio
are shown to work only in tabular case.