# How to express $v_\pi(s)$ in terms of $q_\pi(s,a)$? This is the exercise 3.18 in Sutton and Barto's book.

The task is to express $$v_\pi(s)$$ using $$q_\pi(s,a)$$.

Looking at the diagram above, the value of $$q_\pi(s,a)$$ at $$s$$ for each $$a \in A$$ we take gives us the value function at $$s$$ after taking the action $$a$$ and then following the policy $$\pi$$.

This is probably wrong, but if

$$v_\pi(s) = E_\pi[G_t | S_t = s]$$

and

$$q_\pi(s) = E_\pi[G_t | S_t = s, A_t = a]$$

isn't then $$v_\pi(s)$$ just the expected action value function at $$s$$ over all actions $$a$$ that are given by the policy $$\pi$$, namely

$$v_\pi(s) = E_{a \sim \pi}[q_\pi(s,a) | S_t = s, A_t = a] = \sum_{a \in A}\pi(a|s) q_\pi(s,a)$$?

isn't then $$v_\pi(s)$$ just the expected action value function at $$s$$ over all actions $$a$$ that are given by the policy $$\pi$$, namely

$$v_\pi(s) = E_{a \sim \pi}[q_\pi(s,a) | S_t = s, A_t = a] = \sum_{a \in A}\pi(a|s) q_\pi(s,a)$$?

Yes this is 100% correct.

There is no "trick" to this or deeper thought needed. You have correctly isolated the key part of the MDP description that controls relationship between $$v_{\pi}$$ and $$q_{\pi}$$ in that direction.

Note that for a deterministic policy, with $$\pi(s): \mathcal{S} \rightarrow \mathcal{A}$$ then the relationship is

$$v_\pi(s) = q_\pi(s, \pi(s))$$

The related exercise in the book - expressing $$q_{\pi}$$ in terms of $$v_{\pi}$$ and the MDP characteristics - is more complex because it involves a time step.