The Sutton and Barto reinforcement learning textbook states that

the value of a state under an optimal policy must equal the expected return for the best action from that state.

That is, $$v_*(s) = \max_a q_*(s, a).$$

I am having trouble gaining intuition for this. Since state values can be written as an expectation of the action values under a given policy, I am not sure I see how

$$v_*(s) = \sum_a \pi_*(a|s)q_*(s,a) = \max_a q_*(s, a).$$

I'd appreciate any insights!


You have an optimal policy $\pi_*$, and you are in the state $s$. Because the policy is optimal, it will only give probability to optimal actions. Let's say there are 5 actions $a_1, ..., a_5$ from your current state, and two of those are optimal, $a_2$ and $a_4$. Because they are both optimal, their action values will be equal $q_*(s, a_2) = q_*(s, a_4) = q_\text{optimal}$ and the optimal policy can decide to take either action in any possible ratio with the obvious restriction that the policy has to choose an action. This means that $\pi_*(a_2|s) + \pi_*(a_4|s) = 1$ because all the other actions are sub-optimal and the optimal policy would not take those actions, $\pi_*(a_i|s) = 0$ when $i = 1, 3, 5$. Then, you have:

$$ \begin{align*} v_*(s) &= \sum_i \pi_*(a_i|s)q_*(s, a_i) \\ &= q_\text{optimal} \Big( \pi_*(a_2|s) + \pi_*(a_4|s) \Big) + \sum_{a \in a_1, a_3, a_5} \pi_*(a|s)q_*(s, a) \\ &= q_\text{optimal} \end{align*} $$

The optimal actions are optimal in this scenario because they have the largest action-value, so $q_\text{optimal} = \max_a q_*(s, a)$.

$v_*(s)$ is expected future returns given that you start from state $s$ and follow the optimal policy. $q_*$ is the expected future returns given that you start from state $s$ and take action $a$, then follow the optimal policy. So without the equations above, you can look at $q_*(s, a)$ as a one-step lookahead to evaluate all actions from the current state, and you take the action with the highest action value.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.