# Could you explain steps of proofs of policy improvement theorem in sutton & barto book?

I don't understand one step of the proof. The equality means $$\mathop{\mathbb{E}}[R_{t+2}+\gamma v_\pi(S_{t+2})|S_{t+1},A_{t+1}=\pi'(S_{t+1})]=R_{t+2}+\gamma v_\pi(S_{t+2})$$

Could you explain the detailed steps of why the equality holds?

• You seem to be missing a factor of $\gamma$ on the right hand side, or should remove it from the left hand side, of the equality you are questioning Commented Jun 15, 2023 at 7:10
• Sorry for the confusion. I removed the gamma on the left side Commented Jun 15, 2023 at 12:12

This equality:

$$\mathop{\mathbb{E}}[R_{t+2}+\gamma v_\pi(S_{t+2})|S_{t+1},A_{t+1}=\pi'(S_{t+1})]=R_{t+2}+\gamma v_\pi(S_{t+2})$$

Does not hold in general, and in fact is not well-formed, because $$R_{t+2}$$ and $$S_{t+2}$$ are random variables following a distribution, whilst the expectation of the expression is a single real number.

However, the conversion works in the context of the outer expection, because the conditional part $$S_{t+1},A_{t+1}=\pi'(S_{t+1})$$ matches the outer expectation being $$\mathop{\mathbb{E}_{\pi'}}$$ i.e. the expectation over action choices made using $$\pi'$$. So the inner expectation, and its condition can simply be dropped because the condition on choosing the action using the new improved policy is the same as the outer expectation, just using different notation.

It is important that here the inner expectation is simply added to other terms. If it was multiplied by a variable term that it was dependent upon, the expression could not be unpacked from the inner expectation without affecting the result.

• If so, I think the random variables $S_t$ and $A_t$ are the same as $S_{t+1}$ and $A_{t+1}$. So, the conditions of inner and other expectations are the same, so they can be dropped from inner expectation. Commented Jun 15, 2023 at 23:54
• @teslo No those are different variables, the time step index difference is important. The reason it's the same condition is due to how those variables have been chosen. If it's not clear how they relate to the improved policy, and not the original one, I could add more about that to the answer Commented Jun 16, 2023 at 7:39
• May I ask more details? Commented Jun 20, 2023 at 11:00
• @tesio: Sure, but could you please say where you are stuck, or what you don't understand, so I can add the right details that help you? Commented Jun 20, 2023 at 11:02
• Oh, I can't understand how they relate to the improved policy, and not the original one Commented Jun 20, 2023 at 11:04