I am currently studying the equations 5.2 in Reinforcement Learning An Introduction By Sutton and Barto on page 101.
I want to comprehent the proof by a simple example:

Having only one State with two Actions, each having an Action Value and a Probability of Selection under Policy $\pi$ and $\pi'$

I calculated the action-values for each step with an $\epsilon = 0.1$. The equations hold true for part 1 and 3. But Equation 2 is not the same as Equation 1 as suggested.

Why is $\Sigma q_\pi(s,a)$ in Equation 2 only summing all Action Values without incomporating the Probabilities under policy $\pi$?

Do I need different Action Values for $\pi'$ and $\pi$?

enter image description here


1 Answer 1


Your highlighted equation 2 is incorporating the probabilities under policy $\pi'$ which as stated in your referenced page is the $\epsilon$-greedy policy with respect to $q_{\pi}$, the coefficients ($\frac{\epsilon}{|A(s)|},(1-\epsilon)$) in each summation term are the probabilities.

Here to demonstrate policy improvement theorem you only really need action values of $\pi$ as seen in your referenced derivation, the goal is to compute and compare state values for any generic state $s$ under both policies $\pi$ and $\pi'$.

  • 1
    $\begingroup$ Thank you very much! So the second equation is just a transformation of the first equation to the $\epsilon$-greedy version under the same policy $\pi'$ , which has to be equal or greater than the $\epsilon$-soft equation 3 under policy $\pi$. $\endgroup$
    – bake_thi
    Commented Apr 9 at 8:33
  • $\begingroup$ @bake_thi that sounds right. $\endgroup$
    – cinch
    Commented Apr 9 at 13:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .