Are these two definitions of the state-action value function equivalent?

Question

I have been reading the Sutton and Barto textbook and going through David Silvers UCL lecture videos on YouTube and have a question on the equivalence of two forms of the state-action value function written in terms of the value function.

From Question 3.13 of the textbook I am able to write the state-action value function as $$q_{\pi}(s,a) = \sum_{s',r}p(s',r|s,a)(r + \gamma v_\pi(s')) = \mathbb{E}[r + \gamma v_\pi(s')|s,a]\;.$$ Note that the expectation is not taken with respect to $\pi$ as $\pi$ is the conditional probability of taking action $a$ in state $s$. Now, in David Silver's slides for the Actor-Critic methods of the Policy Gradient lectures, he says that $$\mathbb{E}_{\pi_\theta}[r + \gamma v_{\pi_\theta}(s')|s,a] = q_{\pi_\theta}(s,a)\;.$$

Are these two definitions equivalent (in expectation)?

Answer 1

The definition of the state-action value function is always the same.

Your definition is correct, as $q_{\pi}(s,a)$ is conditioned on $a$, so you don't need to write $q_{\pi}(s,a)$ as an conditional expectation that depends on $\pi$. In fact, the conditional expectation is taken wrt the probability distribution $p(s',r|s,a)$. However, you need the subscript $\pi$ in $v_\pi(s')$ because $v_\pi(s')$ is defined as the expected return by following $\pi$ starting in $s'$.

If you didn't write $q_{\pi}(s,a)$ in terms of $v_\pi(s')$, then you could write $q_{\pi}(s,a)$ as an expectation that depends on $\pi$, because, in that case, $q_{\pi}(s,a)$, is defined as an expectation of the return (after having taken $a$ in $s$), which depends on $\pi$ (see equation 3.3 of Sutton & Barto book, p. 58). Of course, this way of writing $q_{\pi}(s,a)$ is equivalent to writing it in terms of $v_\pi(s')$.

I think David Silver's notation might be an abuse of notation. In his equation, the policy is parametrized by $\theta$, so I think he wants to emphasize that you will estimate the state-action value function (critic) based on $\pi_\theta$ (actor). Alternatively, he uses $\pi_\theta$ as a subscript of $\mathbb{E}_{\pi_\theta}$ to emphasize that the future return starting in $s'$, after having taken action $a$ in $s$, still depends on $\pi_\theta$.