# Why can we use a network to estimate $Q_\pi(s, a)$ in Actor-Critic Method?

According to deep Q learning, we want to learn $$Q^*(s,a)$$, which is the optimal action-value function. It does make sense because we assume there is only one optimal function so the algorithm will converge supposedly.

But when it comes to actor-critic method, we use critic network (also called value network) to estimate $$Q_\pi(s, a)$$. This is what confused me. Since our policy $$\pi$$ will change through time, the target $$Q_\pi(s, a)$$ of value network will also change. What will happen for a network to learn a changing function?

The policy doesn't change over time. That is, the values will change, otherwise we would not be learning anything, but our rules for action selection don't. I.e. we always take action according to the distribution postulated to our current estimate of the policy $$\pi_\theta(a|s)$$, we don't suddenly start taking $$\max_a \pi_\theta(a|s)$$, which would be a true change in policy and would make learning both the actor and the critic unstable.
• @NeilSlater I meant more that the policy doesn't change in the sense that we take actions according to the distribution (which will change) consistently, we don't suddenly start taking e.g. $\max_a \pi_{\theta}(a|s)$. – David Ireland Jul 27 '20 at 9:40