I would like to solve the first question of Exercise 3.19 from Sutton and Barto:

Exercise 3.19 The value of an action, $q_{\pi}(s, a)$, depends on the expected next reward and the expected sum of the remaining rewards. Again we can think of this in terms of a small backup diagram, this one rooted at an action (state–action pair) and branching to the possible next states:

enter image description here

Give the equation corresponding to this intuition and diagram for the action value, $q_{\pi}(s, a)$, in terms of the expected next reward, $R_{t+1}$, and the expected next state value, $v_{\pi}(S_{t+1})$, given that $S_t =s$ and $A_t =a$. This equation should include an expectation but not one conditioned on following the policy.

I'm not sure how to write this equation so that it is not conditioned on $\pi$. It's clear to me that \begin{align*} q_{\pi}(s, a) &= E_{\pi}[G_{t}|S_t=s,A_t=a]\\ & = E_{\pi}[R_{t+1} + \gamma G_{t+1}|S_t=s,A_t=a]\\ & = E_{\pi}[R_{t+1}|S_t=s,A_t=a] + E_{\pi}[\gamma G_{t+1}|S_t=s,A_t=a]. \end{align*}

The second term in the last expression evaluates to (from Exercise 3.13) $$\sum_{s'}\gamma v_{\pi}(s')P(S_{t+1}=s'|A_t=a,S_t=s).$$

Therefore, I'm left with the first term which is $E_{\pi}[R_{t+1}|S_t=s,A_t=a]$.

Question: Can I write the term without the subscript as $$E_{\pi}[R_{t+1}|S_t=s,A_t=a] = E[R_{t+1}|S_t=s,A_t=a]?$$

My reasoning is that the expected reward, given we know what $A_t$ is, does not depend on $\pi$.


1 Answer 1


Question: Can I write it without the subscript? So $$E_{\pi}[R_{t+1}|S_t=s,A_t=a] = E[R_{t+1}|S_t=s,A_t=a]$$

Yes, your reasoning is sound, there is no need to condition the expectation on the policy, because the policy has no influence on the expected value of $R_{t+1}$ given that there is already a condition on $A_t=a$, and $a$ is provided as an argument.

This only works because you have split off the expected reward to evaluate separately. Overall, the value of $q_\pi$ does depend on the policy, and this is shown by using $v_\pi$ in the second term.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .