6 votes
Accepted

Is this proof of $\epsilon$-greedy policy improvement correct?

The weights do sum to one. Note that in the second line where we have $$\frac{\epsilon}{|\mathcal{A}(s)|} \sum_a q_{\pi}(s,a) + (1-\epsilon)\max_aq_{\pi}(s,a) \; ,$$ the sum is over the whole action ...
user avatar
2 votes
Accepted

How do we get from conditional expectation on both state and action to only state in the proof of the Policy Improvement Theorem?

I don't understand how did we get rid of the condition $A_{t}=\pi'(s)$. We don't really, it is just moved into the subscript $\pi'$ in $\mathbb{E}_{\pi'}[]$ - it means the same thing here, that the ...
user avatar
  • 24.5k
2 votes
Accepted

When showing that the policy improvement theorem applies to MC control, why is $q_{\pi_{k}}\left(s, \pi_{k}(s)\right) \geq v_{\pi_{k}}(s)$ true?

You are right that the strict equality $q_\pi(s,\pi(s)) = v_\pi(s)$ is generally true for a deterministic policy $\pi$. The $\geq$ inequality is also correct, of course, and it could be that the ...
user avatar
  • 1,833
2 votes
Accepted

Why and how can the policy and value iteration methods converge to the OPTIMAL point?

These two algorithms converge to the optimal value function because they are instances of the generalization policy iteration, so they iteratively perform one policy evaluation (PE) step followed by ...
user avatar
  • 34.4k

Only top scored, non community-wiki answers of a minimum length are eligible