# PPO advantage estimate - Why does advantage estimate have $r_t+\gamma V(s_{t+1})-V(s_t)$

So I've been looking at this formula for advantage estimate

\begin{equation} \begin{aligned} & \hat{A}_t = \delta_t + (\gamma \lambda)\delta_{t+1} + ... + (\gamma \lambda)^{T-t+1}\delta_{T-1}\\ &\text{where}\ \delta_t = r_t +\gamma V(s_{t+1}) - V(s_t) \end{aligned} \end{equation} where $$r_t$$ is the reward at time $$t$$ and $$V(s_t)$$ is the estimated value of $$s_t$$ and $$\gamma$$ is future discount. Why is $$\delta_t$$ useful? As in why is it useful to calculate reward from time $$t$$ from the discounted estimated reward from that time onward of $$s_t$$ and subtract the value estimate of $$s_t$$

Original PPO Paper

Lets notice, that $$\hat{A}=\delta_t$$ is a unbiased estimate of $$A$$ in a sense, that $$E_{s_{t+1}}[r_t + \gamma V(s_{t+1}) - V(s_t)] = E_{s_{t+1}}[Q(a_t, s_t) - V(s_t)] = A(a_t, s_t)$$ Here we abuse the fact, that $$V(s)$$ is known, but in reality we know only its approximation, so the bias of estimator will be correlated to error of estimation of $$V$$ by $$V_\theta(s)$$. By increasing the trajectory we can minimise the impact of $$V_\theta(s_{t+1})$$, lets write \begin{equation} \begin{aligned} &\hat{A}_t^{(1)}:=\delta^V_t& =r_t + \gamma V(s_{t+1}) - V(s_t) \\ &\hat{A}_t^{(2)}:=\delta^V_t + \gamma\delta^V_{t+1}&=r_t + \gamma r_t + \gamma^2V(s_{t+2}) - V(s_t)\\ &\hat{A}_t^{(3)}:=\delta^V_t + \gamma\delta^V_{t+1}+\gamma^2\delta^V_{t+2}&=r_t + \gamma r_t + \gamma^2r_{t+2}+\gamma^3V(s_{t+3}) - V(s_t)\\ &...&\\ &\hat{A}_t^{(\infty)} := \sum_{i=0}^{\infty} \gamma^i\delta_{t+i}^{V} \end{aligned} \end{equation} The longer the traction, the smaller the term $$\gamma^i$$ at $$V(s_{t+i})$$, therefor the approximation of advantage function is less biased .
It, however, is not perfect, since the variance is increasing with longer path, e.g. $$\hat{A}^{(1)}_t$$ has low variance and high bias, and $$\hat{A}^{(\infty)}_t$$ has low bias, but high variance. The tradeoff can be introduced by taking some $$i < \infty$$ and estimating the A by $$\hat{A}^{(i)}_t$$. But choice of $$i$$ is not evident. By analogy of generalisation for TD($$\lambda$$), described in Sutton's book, chapter 12.8 the other way to introduce the trade-off between bias and variance is to take a weighted sum. In practical case, of course the infinite trajectories are not accessible. So the trajectories of length T are taken into account, which results exactly in $$\hat{A}_t = \sum_{i=0}^{T-(t+1)} (\gamma \lambda)^i \delta^{V_\theta}_{t + i}$$ Where $$\lambda$$ introduces trade off between bias and variance of advantage function estimation.