I have done exercise 3.29 from Sutton and Barto and I'd like to check if it's correct. Here's the exercise:

Rewrite the Bellman equation for the function $q_{\pi}$ in terms of the three argument function $p$ (3.4) and the two-argument function $r$ (3.5).

Here are the 2 functions:

(3.4) :

$$p(s'|s,a) = P(S_t=s'|S_{t-1}=s,A_{t-1}=a)=\sum_{r\in\mathcal{R}}p(s',r|s,a)$$


$$r(s,a) = E[R_t|S_{t-1}=s,A_{t-1}=a]=\sum_{r\in\mathcal{R}}r\sum_{s'\in\mathcal{S}}p(s',r|s,a)$$

Here's my solution. Is this derivation correct?

$$q_\pi(s,a) = \\ E_{\pi}[G_{t+1}|S_t=s,A_t=a]= E_{\pi}[R_{t+1} + \gamma G_{t+1} |S_t=s,A_t=a]= \\ E_[R_{t+1}|S_t=s,A_t=a]+\gamma E_{\pi}[G_{t+1}|S_t=s,A_t=a] = \\ \sum_{s',r}p(s',r|s,a)r+\gamma \sum_{s'}p(s'|s,a)v_{\pi}(s') = \\ r(s,a) + \gamma \sum_{s'} p(s'|s,a) \sum_{a'}q(a'|s')\pi(a'|s')$$


1 Answer 1


Your work all looks correct to me. But I don't have the exercise 3.29 in my book, so I can't read the question for myself. It's not clear to me whether your last line or second to last line should be considered as the answer.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .