Sutton and Barto, in their book (Reinforcement Learning 2nd Edition) begin the discussion of policy improvement by comparing the action value $q_\pi(s, \pi'(s))$ to the state value $v_\pi(s)$.
WhyWhat is the key criteria for policy improvement aintuition behind this comparison of action values to state values?
It seems more intuitivenatural to me to compare $q_\pi(s, \pi'(s))$ and $q_\pi(s, \pi(s))$. I understand that for deterministic policies $q_\pi(s, \pi(s))$ is the same as $v_\pi(s)$ so mathematically it makes no difference. Nonetheless, I’m curious about the intuition behind this comparison. I am wondering as to why a comparison of action values would motivate the theorem poorly compared to the way but perhaps conceptually it is currently presented in the book.does?