I have been recently studying Actor-Critic algorithms, and I ran into the following question.

Let $Q_{\omega}$ be the critic network, and $\pi_{\theta}$ be the actor. It is known that in order to maximize the objective return $J(\theta)$, we follow the gradient direction, which could be estimated as follows $$\nabla_{\theta}J=\mathbb{E}[Q_{\omega}(s,a).\nabla_{\theta}log \pi_{\theta} (a|s)].$$ But if we were to calculate the gradient of $Q^{\pi}$ with respect to $\theta$, what are the possible approaches to do so?

More generally, say we have a network $\phi_{\omega}$ that is trained on data generated from another neural network, say a stochastic actor $\pi_{\theta}$ like in classic reinforcement learning frameworks, how to find the gradient of $\phi_{\omega}$ w.r.t ${\theta}$?

  • $\begingroup$ Gradient would be 0 because it doesn't depend on $\theta$. $\endgroup$
    – Brale
    Apr 30, 2020 at 9:16
  • $\begingroup$ @Brale_ I encourage you to provide a formal answer (even if short) below. If you want, you could say that the derivative is zero by showing some formulas. $\endgroup$
    – nbro
    Apr 30, 2020 at 15:50
  • $\begingroup$ @Brale_ I agree with you, but it's not that straightforward as there wasn't any clue that $\omega$ isn't a function of $\theta$, please refer to (Degris et. al, 2012) for more information. $\endgroup$
    – ASA
    Apr 30, 2020 at 16:52


You must log in to answer this question.