How do we express $q_\pi(s,a)$ as a function of $p(s',r|s,a)$ and $v_\pi(s)$?

Question

The task (exercise 3.13 in the RL book by Sutton and Barto) is to express $q_\pi(s,a)$ as a function of $p(s',r|s,a)$ and $v_\pi(s)$.

$q_\pi(s,a)$ is the action-value function, that states how good it is to be at some state $s$ in the Markov Decision Process (MDP), if at that state, we choose an action $a$, and after that action, the policy $\pi(s,a)$ determines future actions.

Say that we are at some state $s$, and we choose an action $a$. The probability of landing at some other state $s'$ is determined by $p(s',r|s,a)$. Each new state $s'$ then has a state-value function that determines how good is it to be at $s'$ if all future actions are given by the policy $\pi(s',a)$, therefore:

$$q_\pi(s,a) = \sum_{s' \in S} p(s',r|s,a) v_\pi(s')$$

Is this correct?

Answer 1

Not quite. You are missing the reward at time step $t+1$.

The definition you are looking for is (leaving out the $\pi$ subscripts for ease of notation)

$$q(s,a) = \mathbb{E}[R_{t+1} + \gamma v(s') | S_t=s,A_t=a] = \sum_{r,s'}(r +\gamma v(s'))p(s',r|s,a)\;.$$

Because $q(s,a)$ relates to expected returns at time $t$, and returns are defined as $G_t = \sum_{b = 0}^\infty \gamma ^b R_{t+b+1}$, thus $R_{t+1}$ is also a random variable at time $t$ that we need to take expectation with respect to, not just the state that we transition into.