Questions tagged [policy-improvement-theorem]
For questions related to the policy improvement theorem in the context of reinforcement learning.
7
questions
1
vote
1
answer
47
views
Could you explain steps of proofs of policy improvement theorem in sutton & barto book?
I don't understand one step of the proof.
The equality means $\mathop{\mathbb{E}}[R_{t+2}+\gamma v_\pi(S_{t+2})|S_{t+1},A_{t+1}=\pi'(S_{t+1})]=R_{t+2}+\gamma v_\pi(S_{t+2})$
Could you explain the ...
- 185
1
vote
2
answers
1k
views
Why and how can the policy and value iteration methods converge to the OPTIMAL point?
I am reading Reinforcement Learning: An Introduction by Sutton & Barto. According to this textbook, as far as I understood, the authors claim that the policy and value iteration methods converge ...
- 113
0
votes
1
answer
86
views
Policies for which the policy improvement theorem holds
According to Reinforcement Learning (2nd Edition) by Sutton and Barto, the policy improvement theorem states that for any pair of deterministic policies $\pi'$ and $\pi$, if $q_\pi(s,\pi'(s)) \geq v_\...
1
vote
1
answer
83
views
What is the intuition behind comparing action values to state values in the policy improvement theorem?
Sutton and Barto, in their book (Reinforcement Learning 2nd Edition) begin the discussion of policy improvement by comparing the action value $q_\pi(s, \pi'(s))$ to the state value $v_\pi(s)$.
What is ...
1
vote
1
answer
51
views
How do we get from conditional expectation on both state and action to only state in the proof of the Policy Improvement Theorem?
I'm going through Sutton and Barto's book Reinforcement Learning: An Introduction and I'm trying to understand the proof of the Policy Improvement Theorem, presented at page 78 of the physical book.
...
- 563
1
vote
1
answer
75
views
When showing that the policy improvement theorem applies to MC control, why is $q_{\pi_{k}}\left(s, \pi_{k}(s)\right) \geq v_{\pi_{k}}(s)$ true?
When discussing why the policy improvement theorem is true, when we do Monte Carlo control by updating greedily, it says on page 98 of Sutton and Barto's book (2nd edition) that:
$$
\begin{aligned}
...
- 215
6
votes
1
answer
355
views
Is this proof of $\epsilon$-greedy policy improvement correct?
The following paragraph about $\epsilon$-greedy policies can be found at the end of page 100, under section 5.4, of the book "Reinforcement Learning: An Introduction" by Richard Sutton and ...
- 147