0
$\begingroup$

According to deep Q learning, we want to learn $Q^*(s,a)$, which is the optimal action-value function. It does make sense because we assume there is only one optimal function so the algorithm will converge supposedly.

But when it comes to actor-critic method, we use critic network (also called value network) to estimate $Q_\pi(s, a)$. This is what confused me. Since our policy $\pi$ will change through time, the target $Q_\pi(s, a)$ of value network will also change. What will happen for a network to learn a changing function?

Improve this question
$\endgroup$
1
$\begingroup$

The policy doesn't change over time. That is, the values will change, otherwise we would not be learning anything, but our rules for action selection don't. I.e. we always take action according to the distribution postulated to our current estimate of the policy $\pi_\theta(a|s)$, we don't suddenly start taking $\max_a \pi_\theta(a|s)$, which would be a true change in policy and would make learning both the actor and the critic unstable.

Because NN's are able to handle noisy target distributions, this is how they can deal with the changing data. If you think of how Actor-Critic methods work, you would initially start to shift your NN to some unfeasible values (due to random initialisation and the Actor-Critic not having any information about the environment), but as you start to interact with the environment you will start to update the agent towards the 'true' policy.

An analogy in supervised learning would be to have some noisy data which is incorrect and some true data. If you trained your network on the noisy data for a small number of epochs and then never showed it to the network again and trained it solely on the correct data, it would forget it has ever seen the noisy data and only represent the new, true data.

Improve this answer
$\endgroup$
  • $\begingroup$ The rules for derivation of the target policy do not change. However, the target policy does change over time in both cases (Q learning and actor-critic) as estimates are refined. In turn that means the value functions being learned also change over time in both cases the OP is interested in. The reason it works is that NNs can deal with non-stationary target distributions, forgetting the old values. $\endgroup$ – Neil Slater Jul 27 '20 at 9:38
  • $\begingroup$ @NeilSlater I meant more that the policy doesn't change in the sense that we take actions according to the distribution (which will change) consistently, we don't suddenly start taking e.g. $\max_a \pi_{\theta}(a|s)$. $\endgroup$ – David Ireland Jul 27 '20 at 9:40
  • $\begingroup$ Yes, I was justclarifying that in my comment. OP seems to have conflated derivation rule not changing and policy not changing though, so it is important to be clear IMO $\endgroup$ – Neil Slater Jul 27 '20 at 9:41
  • $\begingroup$ right -- I will edit my answer to reflect this. $\endgroup$ – David Ireland Jul 27 '20 at 9:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.