2
$\begingroup$

I have been reading the Sutton and Barto textbook and going through David Silvers UCL lecture videos on YouTube and have a question on the equivalence of two forms of the state-action value function written in terms of the value function.

From Question 3.13 of the textbook I am able to write the state-action value function as $$q_{\pi}(s,a) = \sum_{s',r}p(s',r|s,a)(r + \gamma v_\pi(s')) = \mathbb{E}[r + \gamma v_\pi(s')|s,a]\;.$$ Note that the expectation is not taken with respect to $\pi$ as $\pi$ is the conditional probability of taking action $a$ in state $s$. Now, in David Silver's slides for the Actor-Critic methods of the Policy Gradient lectures, he says that $$\mathbb{E}_{\pi_\theta}[r + \gamma v_{\pi_\theta}(s')|s,a] = q_{\pi_\theta}(s,a)\;.$$

Are these two definitions equivalent (in expectation)?

$\endgroup$
3
$\begingroup$

These two definitions are not exactly the same, even though they have a very similar formulation. David Silver's notation is probably an abuse of notation.

The first difference between those two definitions is that, in the case of David Silver's slides, the policy is parametrized by $\theta$ (i.e. the policy could be represented e.g. by a neural network), while, in the first case, the policy is not parametrized, which is a more general way of writing the definition of the state-action value function, because you do not assume how you will represent the policy.

The second apparent difference (as you have noticed) is that, in the first case, the expectation is taken with respect to transition probability function, while, in the second case, it seems to be taken with respect to the parametrized policy. In the second case, you do not probably know the transition probability function, so you will compute your state-action value function based on your current best estimate of the policy.

Now, as Brale pointed out in the comments (please, read the comments under this answer!), the definition of the state-action value function is always the same, even if you don't know the MDP (because that's a definition and it doesn't matter if you know the transition probability or not). However, in the case of David Silver's slides, by that notation, I think he means he will compute the state-action value function based on the current estimate of the policy.

| improve this answer | |
$\endgroup$
  • $\begingroup$ In the second case expectation is not taken with respect to the policy, it is taken with respect to transition probability distribution just like in the first case. The index $\pi_\theta$ means that value function of $s'$ is with respect to the policy. It would not be a valid definition of $Q$ function if the expectation is with respect to $\pi_\theta$ because the action $a$ is already decided $\endgroup$ – Brale May 7 at 13:46
  • $\begingroup$ @Brale_ But isn't the policy that takes the action $a$? This is in the context of actor-critic methods (which I am not super familiar with). In any case, that's a definition of an expectation for a given $s$ and $a$ (i.e. a conditional expectation). So, I don't understand your last sentence, anyway. Expectations must be taken with respect to some probability distribution or random variable. In this case, it's taken with respect to the probability distribution that the policy represents. $\endgroup$ – nbro May 7 at 13:50
  • $\begingroup$ We are talking about the definition of the $Q$ function, it is always the same, actor-critic context or not. The definition is $Q(s, a) = \sum_{s', r} p(s', r|s, a)(r + \gamma v(s'))$ and not $Q(s,a) =\sum_{a} \pi(a|s)(r + \gamma v(s'))$. You said, if I understood you correctly, that the second one is correct in the second case that OP mentioned ? $\endgroup$ – Brale May 7 at 14:00
  • $\begingroup$ @Brale_ Good point! Now, that you write it explicitly, I have some doubts, also because I am not super familiar with actor-critic methods. Initially, if you look at the history of this answer, I had written that the two definitions are equivalent, but then I realized that there was this index under the expectation. Maybe you're right in saying that the two definitions of the state-action value function are equivalent. $\endgroup$ – nbro May 7 at 14:06
  • 1
    $\begingroup$ Thanks for the help both. I actually realised this myself after discussing with a friend - I got confused with the subscript $\pi$ on the expectations - as @Brale_ points out this is not valid for the definition of the $Q$ function. I realised that whilst we don't know the state-transition probabilities of the MDP explicitly (if we did, then we wouldn't need to use RL methods to solve) we are able to calculate the expectation regardless by approximating it using e.g. Monte Carlo methods, which is what the REINFORCE algorithm does. $\endgroup$ – David Ireland May 7 at 14:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.