I have done exercise 3.29 from Sutton and Barto and I'd like to check if it's correct. Here's the exercise:
Rewrite the four Bellman equations for the four value functions $(v_{\pi},v_*,q_{\pi},q_*)$ in terms of the three argument function $p$ (3.4) and the two-argument function $r$ (3.5).
Here are the 2 functions:
(3.4) :
$$p(s'|s,a) = P(S_t=s'|S_{t-1}=s,A_{t-1}=a)=\sum_{r\in\mathcal{R}}p(s',r|s,a)$$
3.5:
$$r(s,a) = E[R_t|S_{t-1}=s,A_{t-1}=a]=\sum_{r\in\mathcal{R}}r\sum_{s'\in\mathcal{S}}p(s',r|s,a)$$
Here's my solution:
$$v_{\pi}(s) = \\ E_{\pi}[G_t|S_t=s]= \\ E_{\pi}[R_{t+1}+\gamma G_{t+1}|S_t=s]= \\ E[R_{t+1}|S_t=s]+\gamma E_{\pi}[G_{t+1}|S_t=s]= \\ \sum_{s', r,a}p(s',r|s,a)r\pi(a|s)+\gamma \sum_{s',a}p(s'|s,a)v_{\pi}(s')\pi(a|s) = \\ \sum_a\pi(a|s)r(s,a) + \sum_a\pi(a|s)\sum_{s'}v_{\pi}(s')p(s'|s,a)= \\ \sum_a \pi(a|s)[r(s,a)+\gamma \sum_{s'}p(s'|s,a)v_{\pi}(s')]$$
$$v_*(s) = \\ \text{max}_a q_*(s,a) = \text{max }_aE_{\pi}[G_{t}|S_t=s,A_t=a]= \\ \text{max}_aE_{\pi}[R_{t+1}+\gamma G_{t+1}|S_t=s,A_t=a]= \\ \max_a \sum_{s',a}p(s',r|s,a)r+\gamma\sum_{s'}p(s'|s,a)v_*(s')= \\ \text{max}_a r(s,a) + \gamma \sum_{s'}p(s'|s,a)v_*(s')$$
$$q_\pi(s,a) = \\ E_{\pi}[G_{t+1}|S_t=s,A_t=a]= E_{\pi}[R_{t+1} + \gamma G_{t+1} |S_t=s,A_t=a]= \\ E_[R_{t+1}|S_t=s,A_t=a]+\gamma E_{\pi}[G_{t+1}|S_t=s,A_t=a] = \\ \sum_{s',r}p(s',r|s,a)r+\gamma \sum_{s'}p(s'|s,a)v_{\pi}(s') = \\ r(s,a) + \gamma \sum_{s'} p(s'|s,a) \sum_{a'}q(a'|s')\pi(a'|s')$$
$$q_*(s,a)=r(s,a) + \sum_{s'}p(s'|s,a)\text{max}_{a'}q(a'|s')$$
Is this correct?
