Based on my research, I've seen so many on-policy AC approaches that utilise a critic network to estimate the value function $V$. The Bellman equation for the value function is as bellow:

$$ V_\pi(s_t) = \sum_a \pi(a|s_t)\sum_{r, s'}(r+V_\pi(s'))P(s', r|s, a) $$

It makes sense not to have a replay buffer due to the current policy in the formula and the fact that our approach is on-policy. However, I really do not figure out why no one uses a target network to stabilize the training process of the critic, like what we have in DQN, namely the variant published in 2015. Does anyone have an idea for that with probably a citation?

I know that DDPG uses a critic with a fixed target network, but be aware that it is a real off-policy actor-critic. By "real" I mean it is not due to importance sampling.

I have to mention that I can imagine something, but I'm not sure whether it is true or not. If we have a target network, it means we are trying to find a deterministic, optimal in the case of DQN, policy, while we are learning the current policy's data for the actor-critic case with the critic.

  • $\begingroup$ Common boys! Doesn't anyone want the bounty?! The question is not strange. It is something that everyone considers in the papers. $\endgroup$ Nov 24, 2021 at 21:55
  • $\begingroup$ Just to contextualize more, what is an example of an on-policy AC method that does not use the target network that you are referring to? $\endgroup$
    – nbro
    Nov 25, 2021 at 1:31
  • $\begingroup$ @nbro Vanilla on-policy AC does not use target for the critic. Available in textbooks and source codes. $\endgroup$ Nov 25, 2021 at 7:29


You must log in to answer this question.

Browse other questions tagged .