Why isn't a target network used for the critic in on-policy actor-critic methods?

Question

Based on my research, I've seen so many on-policy AC approaches that utilise a critic network to estimate the value function $V$. The Bellman equation for the value function is as bellow:

$$ V_\pi(s_t) = \sum_a \pi(a|s_t)\sum_{r, s'}(r+V_\pi(s'))P(s', r|s, a) $$

It makes sense not to have a replay buffer due to the current policy in the formula and the fact that our approach is on-policy. However, I really do not figure out why no one uses a target network to stabilize the training process of the critic, like what we have in DQN, namely the variant published in 2015. Does anyone have an idea for that with probably a citation?

I know that DDPG uses a critic with a fixed target network, but be aware that it is a real off-policy actor-critic. By "real" I mean it is not due to importance sampling.

I have to mention that I can imagine something, but I'm not sure whether it is true or not. If we have a target network, it means we are trying to find a deterministic, optimal in the case of DQN, policy, while we are learning the current policy's data for the actor-critic case with the critic.

Answer 1

I can't give a conclusive answer, but here's an attempt, largely based on An Empathic Approach to the Problem of Off-policy Temporal-Difference Learning. With function approximation, updating $V_\pi$ for state $s_t$ due to a recorded transition may update state $s^\prime$. Lets say the value estimate for $s_t$ is 1 and the estimate for $s^\prime$ is 2 and we have a linear network, with $\gamma \approx 1$ and learning rate $0.1$. Lets also say we have recorded a transition that gives 0 reward.

In the off-policy setting, when learning from this transition, first we update the value for $s_t$ to 1.1 (and $s^\prime$ to 2.2). The second time the transition is used we update to 1.21 (and 2.42). Repeated off-policy learning can basically cause the learning to diverge as updates get larger. A target network fixes the target value for a longer period, which may mitigate this - the target remains at 2 for a longer time which reduces the updates - but it's still vulnerable to this issue.

In the on-policy setting, once we reach $s^\prime$ we need to learn from a new transition that starts in $s^\prime$, and our current overestimation bias of the value of $s^\prime$ will impact the new error, directly, hopefully counteracting the bias. This is why there are much stronger convergence guarantees in the on policy case for function approximation and bootstrapping. There is some research into target functions for on policy actor-critic but I don't think it has been shown to be more effective, rather the opposite.