The Sutton and Barto reinforcement learning textbook states that
the value of a state under an optimal policy must equal the expected return for the best action from that state.
That is, $$v_*(s) = \max_a q_*(s, a).$$
I am having trouble gaining intuition for this. Since state values can be written as an expectation of the action values under a given policy, I am not sure I see how
$$v_*(s) = \sum_a \pi_*(a|s)q_*(s,a) = \max_a q_*(s, a).$$
I'd appreciate any insights!
You have an optimal policy $\pi_*$, and you are in the state $s$. Because the policy is optimal, it will only give probability to optimal actions. Let's say there are 5 actions $a_1, ..., a_5$ from your current state, and two of those are optimal, $a_2$ and $a_4$. Because they are both optimal, their action values will be equal $q_*(s, a_2) = q_*(s, a_4) = q_\text{optimal}$ and the optimal policy can decide to take either action in any possible ratio with the obvious restriction that the policy has to choose an action. This means that $\pi_*(a_2|s) + \pi_*(a_4|s) = 1$ because all the other actions are sub-optimal and the optimal policy would not take those actions, $\pi_*(a_i|s) = 0$ when $i = 1, 3, 5$. Then, you have:
$$ \begin{align*} v_*(s) &= \sum_i \pi_*(a_i|s)q_*(s, a_i) \\ &= q_\text{optimal} \Big( \pi_*(a_2|s) + \pi_*(a_4|s) \Big) + \sum_{a \in a_1, a_3, a_5} \pi_*(a|s)q_*(s, a) \\ &= q_\text{optimal} \end{align*} $$
The optimal actions are optimal in this scenario because they have the largest action-value, so $q_\text{optimal} = \max_a q_*(s, a)$.
$v_*(s)$ is expected future returns given that you start from state $s$ and follow the optimal policy. $q_*$ is the expected future returns given that you start from state $s$ and take action $a$, then follow the optimal policy. So without the equations above, you can look at $q_*(s, a)$ as a one-step lookahead to evaluate all actions from the current state, and you take the action with the highest action value.
I was also confused when Sutton and Barto claimed in chapter 3, as you stated, that
the value of a state under an optimal policy must equal the expected return for the best action from that state.
However, in chapter 4, they present and prove the "Policy Improvement Theorem" and deduce that
It is a natural extension to consider changes at all states, selecting at each state the action that appears best according to $q_\pi(s, a)$. In other words, to consider the new greedy policy, $\pi'$, given by $\pi'(s) = \underset{a}{argmax} (q_\pi(s, a))$
The fact that $v_∗(s)=\underset{a}{max} \ q_∗(s,a)$ is immediate from there, given that $v_*$ is the optimal value function.
