Why must the value of a state under an optimal policy equal the expected return for the best action from that state?

Question

The Sutton and Barto reinforcement learning textbook states that

the value of a state under an optimal policy must equal the expected return for the best action from that state.

That is, $$v_*(s) = \max_a q_*(s, a).$$

I am having trouble gaining intuition for this. Since state values can be written as an expectation of the action values under a given policy, I am not sure I see how

$$v_*(s) = \sum_a \pi_*(a|s)q_*(s,a) = \max_a q_*(s, a).$$

I'd appreciate any insights!

Answer 1

You have an optimal policy $\pi_*$, and you are in the state $s$. Because the policy is optimal, it will only give probability to optimal actions. Let's say there are 5 actions $a_1, ..., a_5$ from your current state, and two of those are optimal, $a_2$ and $a_4$. Because they are both optimal, their action values will be equal $q_*(s, a_2) = q_*(s, a_4) = q_\text{optimal}$ and the optimal policy can decide to take either action in any possible ratio with the obvious restriction that the policy has to choose an action. This means that $\pi_*(a_2|s) + \pi_*(a_4|s) = 1$ because all the other actions are sub-optimal and the optimal policy would not take those actions, $\pi_*(a_i|s) = 0$ when $i = 1, 3, 5$. Then, you have:

$$ \begin{align*} v_*(s) &= \sum_i \pi_*(a_i|s)q_*(s, a_i) \\ &= q_\text{optimal} \Big( \pi_*(a_2|s) + \pi_*(a_4|s) \Big) + \sum_{a \in a_1, a_3, a_5} \pi_*(a|s)q_*(s, a) \\ &= q_\text{optimal} \end{align*} $$

The optimal actions are optimal in this scenario because they have the largest action-value, so $q_\text{optimal} = \max_a q_*(s, a)$.

$v_*(s)$ is expected future returns given that you start from state $s$ and follow the optimal policy. $q_*$ is the expected future returns given that you start from state $s$ and take action $a$, then follow the optimal policy. So without the equations above, you can look at $q_*(s, a)$ as a one-step lookahead to evaluate all actions from the current state, and you take the action with the highest action value.