6
votes
Accepted
Is this proof of $\epsilon$-greedy policy improvement correct?
The weights do sum to one. Note that in the second line where we have
$$\frac{\epsilon}{|\mathcal{A}(s)|} \sum_a q_{\pi}(s,a) + (1-\epsilon)\max_aq_{\pi}(s,a) \; ,$$
the sum is over the whole action ...
- 4,190
2
votes
Accepted
How do we get from conditional expectation on both state and action to only state in the proof of the Policy Improvement Theorem?
I don't understand how did we get rid of the condition $A_{t}=\pi'(s)$.
We don't really, it is just moved into the subscript $\pi'$ in $\mathbb{E}_{\pi'}[]$ - it means the same thing here, that the ...
- 24.5k
2
votes
Accepted
When showing that the policy improvement theorem applies to MC control, why is $q_{\pi_{k}}\left(s, \pi_{k}(s)\right) \geq v_{\pi_{k}}(s)$ true?
You are right that the strict equality $q_\pi(s,\pi(s)) = v_\pi(s)$ is generally true for a deterministic policy $\pi$.
The $\geq$ inequality is also correct, of course, and it could be that the ...
- 1,833
2
votes
Accepted
Why and how can the policy and value iteration methods converge to the OPTIMAL point?
These two algorithms converge to the optimal value function because
they are instances of the generalization policy iteration, so they iteratively perform one policy evaluation (PE) step followed by ...
- 34.4k
Only top scored, non community-wiki answers of a minimum length are eligible