# How to express $v_\pi(s)$ in terms of $q_\pi(s,a)$?

This is exercise 3.18 in Sutton and Barto's book.

The task is to express $$v_\pi(s)$$ using $$q_\pi(s,a)$$.

Looking at the diagram above, the value of $$q_\pi(s,a)$$ at $$s$$ for each $$a \in A$$ we take gives us the value function at $$s$$ after taking the action $$a$$ and then following the policy $$\pi$$.

This is probably wrong, but if

$$v_\pi(s) = E_\pi[G_t | S_t = s]$$

and

$$q_\pi(s) = E_\pi[G_t | S_t = s, A_t = a]$$

isn't then $$v_\pi(s)$$ just the expected action value function at $$s$$ over all actions $$a$$ that are given by the policy $$\pi$$, namely

$$v_\pi(s) = E_{a \sim \pi}[q_\pi(s,a) | S_t = s, A_t = a] = \sum_{a \in A}\pi(a|s) q_\pi(s,a)$$?

isn't then $$v_\pi(s)$$ just the expected action value function at $$s$$ over all actions $$a$$ that are given by the policy $$\pi$$, namely

$$v_\pi(s) = E_{a \sim \pi}[q_\pi(s,a) | S_t = s, A_t = a] = \sum_{a \in A}\pi(a|s) q_\pi(s,a)$$?

Yes this is 100% correct.

There is no "trick" to this or deeper thought needed. You have correctly isolated the key part of the MDP description that controls relationship between $$v_{\pi}$$ and $$q_{\pi}$$ in that direction.

Note that for a deterministic policy, with $$\pi(s): \mathcal{S} \rightarrow \mathcal{A}$$ then the relationship is

$$v_\pi(s) = q_\pi(s, \pi(s))$$

The related exercise in the book - expressing $$q_{\pi}$$ in terms of $$v_{\pi}$$ and the MDP characteristics - is more complex because it involves a time step.

• It's worth noting that your expression for the deterministic policy is really just a shorter notation for the general case, given that, in the general case, for a deterministic policy, you will multiply $1$ by $q$ and put $a = \pi(s)$ in there for the only possible action, and have $0$ multiplied by $q$ for all other actions. So, the general formula still applies to the deterministic policy case. It's just a shorter notation, but they are mathematically equivalent/equal.
– nbro
Dec 21, 2021 at 11:12