Questions tagged [policy-improvement-theorem]

For questions related to the policy improvement theorem in the context of reinforcement learning.

Filter by
Sorted by
Tagged with
1 vote
1 answer
20 views

Understanding the policy improvement theorem for Monte Carlo Control without Exploring Starts

I am currently studying the equations 5.2 in Reinforcement Learning An Introduction By Sutton and Barto on page 101. I want to comprehent the proof by a simple example: Having only one State with two ...
bake_thi's user avatar
4 votes
1 answer
188 views

Why policy improvement theorem can't be applied in case of function approximation?

Policy improvement theorem is proven as follows: $$v_\pi(s) = \sum_{a \in A} \pi(a|s)q_\pi(a,s) \leq \max_{a \in A} q_\pi(a,s) = q_\pi(\pi'(s), s)$$ What step of the proof does not hold for function ...
Samuel Krempasky's user avatar
2 votes
1 answer
120 views

Could you explain steps of proofs of policy improvement theorem in sutton & barto book?

I don't understand one step of the proof. The equality means $\mathop{\mathbb{E}}[R_{t+2}+\gamma v_\pi(S_{t+2})|S_{t+1},A_{t+1}=\pi'(S_{t+1})]=R_{t+2}+\gamma v_\pi(S_{t+2})$ Could you explain the ...
tesio's user avatar
  • 205
1 vote
2 answers
2k views

Why and how can the policy and value iteration methods converge to the OPTIMAL point?

I am reading Reinforcement Learning: An Introduction by Sutton & Barto. According to this textbook, as far as I understood, the authors claim that the policy and value iteration methods converge ...
Danny_Kim's user avatar
  • 113
0 votes
1 answer
125 views

Policies for which the policy improvement theorem holds

According to Reinforcement Learning (2nd Edition) by Sutton and Barto, the policy improvement theorem states that for any pair of deterministic policies $\pi'$ and $\pi$, if $q_\pi(s,\pi'(s)) \geq v_\...
bonzo_pippinpaddle's user avatar
1 vote
1 answer
86 views

What is the intuition behind comparing action values to state values in the policy improvement theorem?

Sutton and Barto, in their book (Reinforcement Learning 2nd Edition) begin the discussion of policy improvement by comparing the action value $q_\pi(s, \pi'(s))$ to the state value $v_\pi(s)$. What is ...
bonzo_pippinpaddle's user avatar
1 vote
1 answer
56 views

How do we get from conditional expectation on both state and action to only state in the proof of the Policy Improvement Theorem?

I'm going through Sutton and Barto's book Reinforcement Learning: An Introduction and I'm trying to understand the proof of the Policy Improvement Theorem, presented at page 78 of the physical book. ...
Daviiid's user avatar
  • 575
1 vote
1 answer
90 views

When showing that the policy improvement theorem applies to MC control, why is $q_{\pi_{k}}\left(s, \pi_{k}(s)\right) \geq v_{\pi_{k}}(s)$ true?

When discussing why the policy improvement theorem is true, when we do Monte Carlo control by updating greedily, it says on page 98 of Sutton and Barto's book (2nd edition) that: $$ \begin{aligned} ...
Snowball's user avatar
  • 225
6 votes
1 answer
404 views

Is this proof of $\epsilon$-greedy policy improvement correct?

The following paragraph about $\epsilon$-greedy policies can be found at the end of page 100, under section 5.4, of the book "Reinforcement Learning: An Introduction" by Richard Sutton and ...
Jarvis1997's user avatar