1
$\begingroup$

I was reading Proximal Policy Optimization paper. It states following:

The advantage estimator used is:
$\hat{A}_t=-V(s_t)+r_t+\gamma r_{t+1}+...+\gamma^{T-t+1}r_{T-1}+\color{blue}{\gamma^{T-t}}V(s_T) \quad\quad\quad\quad\quad\quad\quad(10)$
where $t$ specifies the time index in $[0, T]$, within a given length-$T$ trajectory segment. Generalizing this choice, we can use a truncated version of generalized advantage estimation, which reduces to Equation (10) when $λ = 1$:
$\hat{A}_t=\delta_t+(\gamma\lambda)\delta_{t+1}+...+(\gamma\lambda)^{T-t+1}\delta_{T-1}\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad(11)$
where, $\delta_t=r_t+\gamma V(s_{t+1})-V(s_t)\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad(12)$

How equation (11) reduces to equation (10). Putting $\lambda=1$ in equation (11), we get:

$\hat{A}_t=\delta_t+\gamma\delta_{t+1}+...+\gamma^{T-t+1}\delta_{T-1}$
Putting equation (12) in equation (11), we get:
$\hat{A}_t$
$=r_t+\gamma V(s_{t+1})-V(s_t) $
$+\gamma[r_{t+1}+\gamma V(s_{t+2})-V(s_{t+1})]+...$
$+\gamma^{T-t+1}[r_{T-1}+\gamma V(s_{T})-V(s_{T-1})]$

$=-V(s_t)+r_t\color{red}{+\gamma V(s_{t+1})} $
$+\gamma r_{t+1}+\gamma^2 V(s_{t+2})\color{red}{-\gamma V(s_{t+1})}+...$
$+\gamma^{T-t+1}r_{T-1}+\color{blue}{\gamma^{T-t+2}} V(s_{T})-V(s_{T-1})$

I understand the terms cancels out. I am not getting the difference in blue colored power of $\gamma$ in last terms. I must have made some stupid mistake.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.