I was reading Proximal Policy Optimization paper. It states following:

The advantage estimator used is:
$\hat{A}_t=-V(s_t)+r_t+\gamma r_{t+1}+...+\gamma^{T-t+1}r_{T-1}+\color{blue}{\gamma^{T-t}}V(s_T) \quad\quad\quad\quad\quad\quad\quad(10)$
where $t$ specifies the time index in $[0, T]$, within a given length-$T$ trajectory segment. Generalizing this choice, we can use a truncated version of generalized advantage estimation, which reduces to Equation (10) when $λ = 1$:
where, $\delta_t=r_t+\gamma V(s_{t+1})-V(s_t)\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad(12)$

How equation (11) reduces to equation (10). Putting $\lambda=1$ in equation (11), we get:

Putting equation (12) in equation (11), we get:
$=r_t+\gamma V(s_{t+1})-V(s_t) $
$+\gamma[r_{t+1}+\gamma V(s_{t+2})-V(s_{t+1})]+...$
$+\gamma^{T-t+1}[r_{T-1}+\gamma V(s_{T})-V(s_{T-1})]$

$=-V(s_t)+r_t\color{red}{+\gamma V(s_{t+1})} $
$+\gamma r_{t+1}+\gamma^2 V(s_{t+2})\color{red}{-\gamma V(s_{t+1})}+...$
$+\gamma^{T-t+1}r_{T-1}+\color{blue}{\gamma^{T-t+2}} V(s_{T})-V(s_{T-1})$

I understand the terms cancels out. I am not getting the difference in blue colored power of $\gamma$ in last terms. I must have made some stupid mistake.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.