# Tag Info

Accepted

### Is this proof of $\epsilon$-greedy policy improvement correct?

The weights do sum to one. Note that in the second line where we have $$\frac{\epsilon}{|\mathcal{A}(s)|} \sum_a q_{\pi}(s,a) + (1-\epsilon)\max_aq_{\pi}(s,a) \; ,$$ the sum is over the whole action ...
• 4,190
Accepted

### How do we get from conditional expectation on both state and action to only state in the proof of the Policy Improvement Theorem?

I don't understand how did we get rid of the condition $A_{t}=\pi'(s)$. We don't really, it is just moved into the subscript $\pi'$ in $\mathbb{E}_{\pi'}[]$ - it means the same thing here, that the ...
• 24.5k
Accepted

### When showing that the policy improvement theorem applies to MC control, why is $q_{\pi_{k}}\left(s, \pi_{k}(s)\right) \geq v_{\pi_{k}}(s)$ true?

You are right that the strict equality $q_\pi(s,\pi(s)) = v_\pi(s)$ is generally true for a deterministic policy $\pi$. The $\geq$ inequality is also correct, of course, and it could be that the ...
• 1,833