I would like to solve the first question of Exercise 3.19 from Sutton and Barto:
Exercise 3.19 The value of an action, $q_{\pi}(s, a)$, depends on the expected next reward and the expected sum of the remaining rewards. Again we can think of this in terms of a small backup diagram, this one rooted at an action (state–action pair) and branching to the possible next states:
Give the equation corresponding to this intuition and diagram for the action value, $q_{\pi}(s, a)$, in terms of the expected next reward, $R_{t+1}$, and the expected next state value, $v_{\pi}(S_{t+1})$, given that $S_t =s$ and $A_t =a$. This equation should include an expectation but not one conditioned on following the policy.
I'm not sure how to write this equation so that it is not conditioned on $\pi$. It's clear to me that \begin{align*} q_{\pi}(s, a) &= E_{\pi}[G_{t}|S_t=s,A_t=a]\\ & = E_{\pi}[R_{t+1} + \gamma G_{t+1}|S_t=s,A_t=a]\\ & = E_{\pi}[R_{t+1}|S_t=s,A_t=a] + E_{\pi}[\gamma G_{t+1}|S_t=s,A_t=a]. \end{align*}
The second term in the last expression evaluates to (from Exercise 3.13) $$\sum_{s'}\gamma v_{\pi}(s')P(S_{t+1}=s'|A_t=a,S_t=s).$$
Therefore, I'm left with the first term which is $E_{\pi}[R_{t+1}|S_t=s,A_t=a]$.
Question: Can I write the term without the subscript as $$E_{\pi}[R_{t+1}|S_t=s,A_t=a] = E[R_{t+1}|S_t=s,A_t=a]?$$
My reasoning is that the expected reward, given we know what $A_t$ is, does not depend on $\pi$.
