3
$\begingroup$

enter image description here

This is exercise 3.18 in Sutton and Barto's book.

The task is to express $v_\pi(s)$ using $q_\pi(s,a)$.

Looking at the diagram above, the value of $q_\pi(s,a)$ at $s$ for each $a \in A$ we take gives us the value function at $s$ after taking the action $a$ and then following the policy $\pi$.

This is probably wrong, but if

$$v_\pi(s) = E_\pi[G_t | S_t = s]$$

and

$$q_\pi(s) = E_\pi[G_t | S_t = s, A_t = a]$$

isn't then $v_\pi(s)$ just the expected action value function at $s$ over all actions $a$ that are given by the policy $\pi$, namely

$$v_\pi(s) = E_{a \sim \pi}[q_\pi(s,a) | S_t = s, A_t = a] = \sum_{a \in A}\pi(a|s) q_\pi(s,a)$$?

$\endgroup$

1 Answer 1

4
$\begingroup$

isn't then $v_\pi(s)$ just the expected action value function at $s$ over all actions $a$ that are given by the policy $\pi$, namely

$v_\pi(s) = E_{a \sim \pi}[q_\pi(s,a) | S_t = s, A_t = a] = \sum_{a \in A}\pi(a|s) q_\pi(s,a)$?

Yes this is 100% correct.

There is no "trick" to this or deeper thought needed. You have correctly isolated the key part of the MDP description that controls relationship between $v_{\pi}$ and $q_{\pi}$ in that direction.

Note that for a deterministic policy, with $\pi(s): \mathcal{S} \rightarrow \mathcal{A}$ then the relationship is

$$v_\pi(s) = q_\pi(s, \pi(s))$$

The related exercise in the book - expressing $q_{\pi}$ in terms of $v_{\pi}$ and the MDP characteristics - is more complex because it involves a time step.

$\endgroup$
1
  • $\begingroup$ It's worth noting that your expression for the deterministic policy is really just a shorter notation for the general case, given that, in the general case, for a deterministic policy, you will multiply $1$ by $q$ and put $a = \pi(s)$ in there for the only possible action, and have $0$ multiplied by $q$ for all other actions. So, the general formula still applies to the deterministic policy case. It's just a shorter notation, but they are mathematically equivalent/equal. $\endgroup$
    – nbro
    Dec 21, 2021 at 11:12

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .