# Understanding the policy improvement theorem for Monte Carlo Control without Exploring Starts

I am currently studying the equations 5.2 in Reinforcement Learning An Introduction By Sutton and Barto on page 101.
I want to comprehent the proof by a simple example:

Having only one State with two Actions, each having an Action Value and a Probability of Selection under Policy $$\pi$$ and $$\pi'$$

I calculated the action-values for each step with an $$\epsilon = 0.1$$. The equations hold true for part 1 and 3. But Equation 2 is not the same as Equation 1 as suggested.

Why is $$\Sigma q_\pi(s,a)$$ in Equation 2 only summing all Action Values without incomporating the Probabilities under policy $$\pi$$?

Do I need different Action Values for $$\pi'$$ and $$\pi$$?

Your highlighted equation 2 is incorporating the probabilities under policy $$\pi'$$ which as stated in your referenced page is the $$\epsilon$$-greedy policy with respect to $$q_{\pi}$$, the coefficients ($$\frac{\epsilon}{|A(s)|},(1-\epsilon)$$) in each summation term are the probabilities.
Here to demonstrate policy improvement theorem you only really need action values of $$\pi$$ as seen in your referenced derivation, the goal is to compute and compare state values for any generic state $$s$$ under both policies $$\pi$$ and $$\pi'$$.
• Thank you very much! So the second equation is just a transformation of the first equation to the $\epsilon$-greedy version under the same policy $\pi'$ , which has to be equal or greater than the $\epsilon$-soft equation 3 under policy $\pi$. Commented Apr 9 at 8:33