edited tags
Link
nbro
  • 31.2k
  • 8
  • 65
  • 130
deleted 235 characters in body; edited title
Source Link

Intuition What is the intuition behind comparing action values to state values in the policy improvement theorem for deterministic policies in reinforcement learning?

Sutton and Barto, in their book (Reinforcement Learning 2nd Edition) begin the discussion of policy improvement by comparing the action value $q_\pi(s, \pi'(s))$ to the state value $v_\pi(s)$.

WhyWhat is the key criteria for policy improvement aintuition behind this comparison of action values to state values?

It seems more intuitivenatural to me to compare $q_\pi(s, \pi'(s))$ and $q_\pi(s, \pi(s))$. I understand that for deterministic policies $q_\pi(s, \pi(s))$ is the same as $v_\pi(s)$ so mathematically it makes no difference. Nonetheless, I’m curious about the intuition behind this comparison. I am wondering as to why a comparison of action values would motivate the theorem poorly compared to the way but perhaps conceptually it is currently presented in the book.does?

Intuition behind the policy improvement theorem for deterministic policies in reinforcement learning

Sutton and Barto, in their book (Reinforcement Learning 2nd Edition) begin the discussion of policy improvement by comparing the action value $q_\pi(s, \pi'(s))$ to the state value $v_\pi(s)$.

Why is the key criteria for policy improvement a comparison of action values to state values?

It seems more intuitive to me to compare $q_\pi(s, \pi'(s))$ and $q_\pi(s, \pi(s))$. I understand that for deterministic policies $q_\pi(s, \pi(s))$ is the same as $v_\pi(s)$ so mathematically it makes no difference. Nonetheless, I’m curious about the intuition behind this comparison. I am wondering as to why a comparison of action values would motivate the theorem poorly compared to the way it is currently presented in the book.

What is the intuition behind comparing action values to state values in the policy improvement theorem?

Sutton and Barto, in their book (Reinforcement Learning 2nd Edition) begin the discussion of policy improvement by comparing the action value $q_\pi(s, \pi'(s))$ to the state value $v_\pi(s)$.

What is the intuition behind this comparison?

It seems more natural to me to compare $q_\pi(s, \pi'(s))$ and $q_\pi(s, \pi(s))$. I understand that for deterministic policies $q_\pi(s, \pi(s))$ is the same as $v_\pi(s)$ so mathematically it makes no difference but perhaps conceptually it does?

deleted 33 characters in body; edited tags
Source Link
nbro
  • 31.2k
  • 8
  • 65
  • 130

Sutton and Barto, in their book (Reinforcement Learning 2nd Edition) begin the discussion of policy improvement by comparing the action value $q_\pi(s, \pi'(s))$ to the state value $v_\pi(s)$.

Why is the key criteria for policy improvement a comparison of action values to state values?

It seems more intuitive to me to compare $q_\pi(s, \pi'(s))$ and $q_\pi(s, \pi(s))$. I understand that for deterministic policies $q_\pi(s, \pi(s))$ is the same as $v_\pi(s)$ so mathematically it makes no difference. Nonetheless, I’m curious about the intuition behind this comparison. I am wondering as to why a comparison of action values would motivate the theorem poorly compared to the way it is currently presented in the book.

I'd appreciate any insights!

Sutton and Barto, in their book (Reinforcement Learning 2nd Edition) begin the discussion of policy improvement by comparing the action value $q_\pi(s, \pi'(s))$ to the state value $v_\pi(s)$.

Why is the key criteria for policy improvement a comparison of action values to state values?

It seems more intuitive to me to compare $q_\pi(s, \pi'(s))$ and $q_\pi(s, \pi(s))$. I understand that for deterministic policies $q_\pi(s, \pi(s))$ is the same as $v_\pi(s)$ so mathematically it makes no difference. Nonetheless, I’m curious about the intuition behind this comparison. I am wondering as to why a comparison of action values would motivate the theorem poorly compared to the way it is currently presented in the book.

I'd appreciate any insights!

Sutton and Barto, in their book (Reinforcement Learning 2nd Edition) begin the discussion of policy improvement by comparing the action value $q_\pi(s, \pi'(s))$ to the state value $v_\pi(s)$.

Why is the key criteria for policy improvement a comparison of action values to state values?

It seems more intuitive to me to compare $q_\pi(s, \pi'(s))$ and $q_\pi(s, \pi(s))$. I understand that for deterministic policies $q_\pi(s, \pi(s))$ is the same as $v_\pi(s)$ so mathematically it makes no difference. Nonetheless, I’m curious about the intuition behind this comparison. I am wondering as to why a comparison of action values would motivate the theorem poorly compared to the way it is currently presented in the book.

deleted 14 characters in body
Source Link
hanugm
  • 2.8k
  • 2
  • 8
  • 24
Loading
added 2 characters in body
Source Link
Loading
Source Link
Loading