Could you explain steps of proofs of policy improvement theorem in sutton & barto book?

Question

I don't understand one step of the proof. The equality means $\mathop{\mathbb{E}}[R_{t+2}+\gamma v_\pi(S_{t+2})|S_{t+1},A_{t+1}=\pi'(S_{t+1})]=R_{t+2}+\gamma v_\pi(S_{t+2})$

Could you explain the detailed steps of why the equality holds?

Answer 1

This equality:

$\mathop{\mathbb{E}}[R_{t+2}+\gamma v_\pi(S_{t+2})|S_{t+1},A_{t+1}=\pi'(S_{t+1})]=R_{t+2}+\gamma v_\pi(S_{t+2})$

Does not hold in general, and in fact is not well-formed, because $R_{t+2}$ and $S_{t+2}$ are random variables following a distribution, whilst the expectation of the expression is a single real number.

However, the conversion works in the context of the outer expection, because the conditional part $S_{t+1},A_{t+1}=\pi'(S_{t+1})$ matches the outer expectation being $\mathop{\mathbb{E}_{\pi'}}$ i.e. the expectation over action choices made using $\pi'$. So the inner expectation, and its condition can simply be dropped because the condition on choosing the action using the new improved policy is the same as the outer expectation, just using different notation.

It is important that here the inner expectation is simply added to other terms. If it was multiplied by a variable term that it was dependent upon, the expression could not be unpacked from the inner expectation without affecting the result.