# $E_{\pi}[R_{t+1}|S_t=s,A_t=a] = E[R_{t+1}|S_t=s,A_t=a]$?

I would like to solve the first question of Exercise 3.19 from Sutton and Barto:

Exercise 3.19 The value of an action, $$q_{\pi}(s, a)$$, depends on the expected next reward and the expected sum of the remaining rewards. Again we can think of this in terms of a small backup diagram, this one rooted at an action (state–action pair) and branching to the possible next states: Give the equation corresponding to this intuition and diagram for the action value, $$q_{\pi}(s, a)$$, in terms of the expected next reward, $$R_{t+1}$$, and the expected next state value, $$v_{\pi}(S_{t+1})$$, given that $$S_t =s$$ and $$A_t =a$$. This equation should include an expectation but not one conditioned on following the policy.

I'm not sure how to write this equation so that it is not conditioned on $$\pi$$. It's clear to me that \begin{align*} q_{\pi}(s, a) &= E_{\pi}[G_{t}|S_t=s,A_t=a]\\ & = E_{\pi}[R_{t+1} + \gamma G_{t+1}|S_t=s,A_t=a]\\ & = E_{\pi}[R_{t+1}|S_t=s,A_t=a] + E_{\pi}[\gamma G_{t+1}|S_t=s,A_t=a]. \end{align*}

The second term in the last expression evaluates to (from Exercise 3.13) $$\sum_{s'}\gamma v_{\pi}(s')P(S_{t+1}=s'|A_t=a,S_t=s).$$

Therefore, I'm left with the first term which is $$E_{\pi}[R_{t+1}|S_t=s,A_t=a]$$.

Question: Can I write the term without the subscript as $$E_{\pi}[R_{t+1}|S_t=s,A_t=a] = E[R_{t+1}|S_t=s,A_t=a]?$$

My reasoning is that the expected reward, given we know what $$A_t$$ is, does not depend on $$\pi$$.

Question: Can I write it without the subscript? So $$E_{\pi}[R_{t+1}|S_t=s,A_t=a] = E[R_{t+1}|S_t=s,A_t=a]$$
Yes, your reasoning is sound, there is no need to condition the expectation on the policy, because the policy has no influence on the expected value of $$R_{t+1}$$ given that there is already a condition on $$A_t=a$$, and $$a$$ is provided as an argument.
This only works because you have split off the expected reward to evaluate separately. Overall, the value of $$q_\pi$$ does depend on the policy, and this is shown by using $$v_\pi$$ in the second term.