# Connection between the Bellman equation for the action value function $q_\pi(s,a)$ and expressing $q_\pi(s,a) = q_\pi(s, a,v_\pi(s'))$

When deriving the Bellman equation for $$q_\pi(s,a)$$, we have

$$q_\pi(s,a) = E_\pi[G_t | S_t = s, A_t = a] = E_\pi[R_{t+1} + \gamma G_{t+1} | S_t = s, A_t = a]$$ (1)

This is what is confusing me, at this point, for the Bellman equation for $$q_\pi(s,a)$$, we write $$G_{t+1}$$ as an expected value, conditioned on $$s'$$ and $$a'$$ of the action value function at $$s'$$, otherwise, there is no recursion with respect to $$q_\pi(s,a)$$, and therefore no Bellman equation. Namely,

$$= \sum_{a \in A} \pi(a |s) \sum_{s' \in S} \sum_{r \in R} p(s',r|s,a)(r + \gamma E_\pi[G_{t+1}|S_{t+1} = s', A_{t+1} = a'])$$ (2)

which introduces the recursion of $$q$$,

$$= \sum_{a \in A} \pi(a |s) \sum_{s' \in S} \sum_{r \in R} p(s',r|s,a)(r + \gamma q_\pi(s',a'))$$ (3)

which should be the Bellman equation for $$q_\pi(s,a)$$, right?

On the other hand, when connecting $$q_\pi(s,a)$$ with $$v_\pi(s')$$, in this answer, I believe this is done

$$q_\pi(s,a) = \sum_{a\in A} \pi(a |s) \sum_{s' \in S}\sum_{r \in R} p(s',r|s,a)(r + \gamma E_{\pi}[G_{t+1} | S_{t+1} = s'])$$ (4)

$$q_\pi(s,a) = \sum_{a\in A} \pi(a |s) \sum_{s' \in S}\sum_{r \in R} p(s',r|s,a)(r + \gamma v_\pi(s'))$$ (5)

Is the difference between using the expectation $$E_{\pi}[G_{t+1} | S_{t+1} = s', A_{t+1} = a']$$ in (3) and the expectation $$E_{\pi}[G_{t+1} | S_{t+1} = s']$$ in $$(4)$$ simply the difference in how we choose to express the expected return $$G_{t+1}$$ at $$s'$$ in the definition of $$q_\pi(s,a)$$?

In $$3$$, we express the total return at $$s'$$ using the action value function

leading to the recursion and the Bellman equation, and in $$4$$, the total return is expressed at $$s'$$ using the value function

leading to $$q_\pi(s,a) = q_\pi(s,a,v_\pi(s'))$$?

## 1 Answer

Your understanding of the Bellman equation is not quite right. The state-action value function is defined as the expected (discounted) returns when taking action $$a$$ in state $$s$$. Now, in your equation (2) you have conditioned on taking action $$a'$$ in the inner expectation - this is not what happens in the state-action value function, you do not condition on knowing $$A_{t+1}$$, it is chosen according to the policy $$\pi$$ as per the definition of a bellman equation.

If you want to see a 'recursion' between state action value functions, note that

$$v_\pi(s) = \sum_a \pi(a|s)q_\pi(s,a)\;,$$

Your equation (5) is incorrect -- you need to drop the outter sum over $$a$$ as we have conditioned on knowing $$a$$. I will drop the $$\pi$$ subscripts for ease on notation, and we can see a 'recursion' for state-action value functions as:

$$q(s,a) = \sum_{s',r}p(s',r|s,a)\left(r + \gamma \left[\sum_{a'} \pi(a'|s')q(s',a')\right]\right)\;.$$