0
$\begingroup$

I have done exercise 3.29 from Sutton and Barto and I'd like to check if it's correct. Here's the exercise:

Rewrite the four Bellman equations for the four value functions $(v_{\pi},v_*,q_{\pi},q_*)$ in terms of the three argument function $p$ (3.4) and the two-argument function $r$ (3.5).

Here are the 2 functions:

(3.4) :

$$p(s'|s,a) = P(S_t=s'|S_{t-1}=s,A_{t-1}=a)=\sum_{r\in\mathcal{R}}p(s',r|s,a)$$

3.5:

$$r(s,a) = E[R_t|S_{t-1}=s,A_{t-1}=a]=\sum_{r\in\mathcal{R}}r\sum_{s'\in\mathcal{S}}p(s',r|s,a)$$

Here's my solution:

$$v_{\pi}(s) = \\ E_{\pi}[G_t|S_t=s]= \\ E_{\pi}[R_{t+1}+\gamma G_{t+1}|S_t=s]= \\ E[R_{t+1}|S_t=s]+\gamma E_{\pi}[G_{t+1}|S_t=s]= \\ \sum_{s', r,a}p(s',r|s,a)r\pi(a|s)+\gamma \sum_{s',a}p(s'|s,a)v_{\pi}(s')\pi(a|s) = \\ \sum_a\pi(a|s)r(s,a) + \sum_a\pi(a|s)\sum_{s'}v_{\pi}(s')p(s'|s,a)= \\ \sum_a \pi(a|s)[r(s,a)+\gamma \sum_{s'}p(s'|s,a)v_{\pi}(s')]$$

$$v_*(s) = \\ \text{max}_a q_*(s,a) = \text{max }_aE_{\pi}[G_{t}|S_t=s,A_t=a]= \\ \text{max}_aE_{\pi}[R_{t+1}+\gamma G_{t+1}|S_t=s,A_t=a]= \\ \max_a \sum_{s',a}p(s',r|s,a)r+\gamma\sum_{s'}p(s'|s,a)v_*(s')= \\ \text{max}_a r(s,a) + \gamma \sum_{s'}p(s'|s,a)v_*(s')$$

$$q_\pi(s,a) = \\ E_{\pi}[G_{t+1}|S_t=s,A_t=a]= E_{\pi}[R_{t+1} + \gamma G_{t+1} |S_t=s,A_t=a]= \\ E_[R_{t+1}|S_t=s,A_t=a]+\gamma E_{\pi}[G_{t+1}|S_t=s,A_t=a] = \\ \sum_{s',r}p(s',r|s,a)r+\gamma \sum_{s'}p(s'|s,a)v_{\pi}(s') = \\ r(s,a) + \gamma \sum_{s'} p(s'|s,a) \sum_{a'}q(a'|s')\pi(a'|s')$$

$$q_*(s,a)=r(s,a) + \sum_{s'}p(s'|s,a)\text{max}_{a'}q(a'|s')$$

Is this correct?

$\endgroup$
3
  • $\begingroup$ Can you please edit this post to focus on one bellman equation at a time and write the other bellman equations in other posts? Moreover, please, don't just ask "is this correct?". Maybe try to argue why or not. $\endgroup$
    – nbro
    Commented Jul 5, 2022 at 18:48
  • $\begingroup$ @nbro as you said in a previous comment, I asked separate questions for each equation. The one that you referenced is only 1 of them. Shall I delete this question or? $\endgroup$
    – user
    Commented Jul 8, 2022 at 7:06
  • $\begingroup$ Yes, exactly, you should delete it. I voted to close this as a duplicate of only 1 because I cannot do it for multiples ones. Thank you. $\endgroup$
    – nbro
    Commented Jul 8, 2022 at 21:23

0