I have done exercise 3.29 from Sutton and Barto and I'd like to check if it's correct. Here's the exercise:
Rewrite the Bellman equation for the function $q_{\pi}$ in terms of the three argument function $p$ (3.4) and the two-argument function $r$ (3.5).
Here are the 2 functions:
(3.4) :
$$p(s'|s,a) = P(S_t=s'|S_{t-1}=s,A_{t-1}=a)=\sum_{r\in\mathcal{R}}p(s',r|s,a)$$
3.5:
$$r(s,a) = E[R_t|S_{t-1}=s,A_{t-1}=a]=\sum_{r\in\mathcal{R}}r\sum_{s'\in\mathcal{S}}p(s',r|s,a)$$
Here's my solution. Is this derivation correct?
$$q_\pi(s,a) = \\ E_{\pi}[G_{t+1}|S_t=s,A_t=a]= E_{\pi}[R_{t+1} + \gamma G_{t+1} |S_t=s,A_t=a]= \\ E_[R_{t+1}|S_t=s,A_t=a]+\gamma E_{\pi}[G_{t+1}|S_t=s,A_t=a] = \\ \sum_{s',r}p(s',r|s,a)r+\gamma \sum_{s'}p(s'|s,a)v_{\pi}(s') = \\ r(s,a) + \gamma \sum_{s'} p(s'|s,a) \sum_{a'}q(a'|s')\pi(a'|s')$$