# What is the gradient of the Q function with respect to the policy's parameters?

I have been recently studying Actor-Critic algorithms, and I ran into the following question.

Let $$Q_{\omega}$$ be the critic network, and $$\pi_{\theta}$$ be the actor. It is known that in order to maximize the objective return $$J(\theta)$$, we follow the gradient direction, which could be estimated as follows $$\nabla_{\theta}J=\mathbb{E}[Q_{\omega}(s,a).\nabla_{\theta}log \pi_{\theta} (a|s)].$$ But if we were to calculate the gradient of $$Q^{\pi}$$ with respect to $$\theta$$, what are the possible approaches to do so?

More generally, say we have a network $$\phi_{\omega}$$ that is trained on data generated from another neural network, say a stochastic actor $$\pi_{\theta}$$ like in classic reinforcement learning frameworks, how to find the gradient of $$\phi_{\omega}$$ w.r.t $${\theta}$$?

• Gradient would be 0 because it doesn't depend on $\theta$. – Brale Apr 30 at 9:16
• @Brale_ I encourage you to provide a formal answer (even if short) below. If you want, you could say that the derivative is zero by showing some formulas. – nbro Apr 30 at 15:50
• @Brale_ I agree with you, but it's not that straightforward as there wasn't any clue that $\omega$ isn't a function of $\theta$, please refer to (Degris et. al, 2012) for more information. – Ammora Apr 30 at 16:52