When deriving the Bellman equation for $q_\pi(s,a)$, we have

$q_\pi(s,a) = E_\pi[G_t | S_t = s, A_t = a] = E_\pi[R_{t+1} + \gamma G_{t+1} | S_t = s, A_t = a]$ (1)

This is what is confusing me, at this point, for the Bellman equation for $q_\pi(s,a)$, we write $G_{t+1}$ as an expected value, conditioned on $s'$ and $a'$ of the action value function at $s'$, otherwise, there is no recursion with respect to $q_\pi(s,a)$, and therefore no Bellman equation. Namely,

$ = \sum_{a \in A} \pi(a |s) \sum_{s' \in S} \sum_{r \in R} p(s',r|s,a)(r + \gamma E_\pi[G_{t+1}|S_{t+1} = s', A_{t+1} = a'])$ (2)

which introduces the recursion of $q$,

$ = \sum_{a \in A} \pi(a |s) \sum_{s' \in S} \sum_{r \in R} p(s',r|s,a)(r + \gamma q_\pi(s',a'))$ (3)

which should be the Bellman equation for $q_\pi(s,a)$, right?

On the other hand, when connecting $q_\pi(s,a)$ with $v_\pi(s')$, in this answer, I believe this is done

$q_\pi(s,a) = \sum_{a\in A} \pi(a |s) \sum_{s' \in S}\sum_{r \in R} p(s',r|s,a)(r + \gamma E_{\pi}[G_{t+1} | S_{t+1} = s'])$ (4)

$q_\pi(s,a) = \sum_{a\in A} \pi(a |s) \sum_{s' \in S}\sum_{r \in R} p(s',r|s,a)(r + \gamma v_\pi(s'))$ (5)

Is the difference between using the expectation $E_{\pi}[G_{t+1} | S_{t+1} = s', A_{t+1} = a']$ in (3) and the expectation $E_{\pi}[G_{t+1} | S_{t+1} = s']$ in $(4)$ simply the difference in how we choose to express the expected return $G_{t+1}$ at $s'$ in the definition of $q_\pi(s,a)$?

In $3$, we express the total return at $s'$ using the action value function

enter image description here

leading to the recursion and the Bellman equation, and in $4$, the total return is expressed at $s'$ using the value function

enter image description here

leading to $q_\pi(s,a) = q_\pi(s,a,v_\pi(s'))$?


Your understanding of the Bellman equation is not quite right. The state-action value function is defined as the expected (discounted) returns when taking action $a$ in state $s$. Now, in your equation (2) you have conditioned on taking action $a'$ in the inner expectation - this is not what happens in the state-action value function, you do not condition on knowing $A_{t+1}$, it is chosen according to the policy $\pi$ as per the definition of a bellman equation.

If you want to see a 'recursion' between state action value functions, note that

$$v_\pi(s) = \sum_a \pi(a|s)q_\pi(s,a)\;,$$

Your equation (5) is incorrect -- you need to drop the outter sum over $a$ as we have conditioned on knowing $a$. I will drop the $\pi$ subscripts for ease on notation, and we can see a 'recursion' for state-action value functions as:

$$q(s,a) = \sum_{s',r}p(s',r|s,a)\left(r + \gamma \left[\sum_{a'} \pi(a'|s')q(s',a')\right]\right)\;.$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.