# Return to Question

edited tags
nbro
• 31.2k
• 8
• 65
• 130
deleted 235 characters in body; edited title

# Intuition What is the intuition behind comparing action values to state values in the policy improvement theorem for deterministic policies in reinforcement learning?

Sutton and Barto, in their book (Reinforcement Learning 2nd Edition) begin the discussion of policy improvement by comparing the action value $$q_\pi(s, \pi'(s))$$ to the state value $$v_\pi(s)$$.

WhyWhat is the key criteria for policy improvement aintuition behind this comparison of action values to state values?

It seems more intuitivenatural to me to compare $$q_\pi(s, \pi'(s))$$ and $$q_\pi(s, \pi(s))$$. I understand that for deterministic policies $$q_\pi(s, \pi(s))$$ is the same as $$v_\pi(s)$$ so mathematically it makes no difference. Nonetheless, I’m curious about the intuition behind this comparison. I am wondering as to why a comparison of action values would motivate the theorem poorly compared to the way but perhaps conceptually it is currently presented in the book.does?

deleted 33 characters in body; edited tags
nbro
• 31.2k
• 8
• 65
• 130

Sutton and Barto, in their book (Reinforcement Learning 2nd Edition) begin the discussion of policy improvement by comparing the action value $$q_\pi(s, \pi'(s))$$ to the state value $$v_\pi(s)$$.

Why is the key criteria for policy improvement a comparison of action values to state values?

It seems more intuitive to me to compare $$q_\pi(s, \pi'(s))$$ and $$q_\pi(s, \pi(s))$$. I understand that for deterministic policies $$q_\pi(s, \pi(s))$$ is the same as $$v_\pi(s)$$ so mathematically it makes no difference. Nonetheless, I’m curious about the intuition behind this comparison. I am wondering as to why a comparison of action values would motivate the theorem poorly compared to the way it is currently presented in the book.

I'd appreciate any insights!

deleted 14 characters in body
hanugm
• 2.8k
• 2
• 8
• 24
Loading
added 2 characters in body
Loading
Loading