1
$\begingroup$

Excerpt from Sutton and Barto:

The expected value of all $n$-step returns is guaranteed to improve in a certain way over the current value function as an approximation to the true value function. For any $V$, the expected value of the $n$-step return using $V$ is guaranteed to be a better estimate of $V^{\pi}$ than $V$ is, in a worst-state sense. That is, the worst error under the new estimate is guaranteed to be less than or equal to $\gamma^n$ times the worst error under $V$:

enter image description here

My question is why did they add the $\gamma^n$ in the RHS of the equation? My understanding is that in the LHS we use the $E_{\pi}$ expectation term to get the estimate of $V^{\pi}(s)$, then why is the difference in the second term diminished by $\gamma^n$. Can it be explained informally or is it a result of a proof?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.