Lets notice, that $\hat{A}=\delta_t$ is a unbiased estimate of $A$ in a sense, that
$$
E_{s_{t+1}}[r_t + \gamma V(s_{t+1}) - V(s_t)] = E_{s_{t+1}}[Q(a_t, s_t) - V(s_t)] = A(a_t, s_t)
$$
Here we abuse the fact, that $V(s)$ is known, but in reality we know only its approximation, so the bias of estimator will be correlated to error of estimation of $V$ by $V_\theta(s)$. By increasing the trajectory we can minimise the impact of $V_\theta(s_{t+1})$, lets write
\begin{equation}
\begin{aligned}
&\hat{A}_t^{(1)}:=\delta^V_t& =r_t + \gamma V(s_{t+1}) - V(s_t) \\
&\hat{A}_t^{(2)}:=\delta^V_t + \gamma\delta^V_{t+1}&=r_t + \gamma r_t + \gamma^2V(s_{t+2}) - V(s_t)\\
&\hat{A}_t^{(3)}:=\delta^V_t + \gamma\delta^V_{t+1}+\gamma^2\delta^V_{t+2}&=r_t + \gamma r_t + \gamma^2r_{t+2}+\gamma^3V(s_{t+3}) - V(s_t)\\
&...&\\
&\hat{A}_t^{(\infty)} := \sum_{i=0}^{\infty} \gamma^i\delta_{t+i}^{V}
\end{aligned}
\end{equation}
The longer the traction, the smaller the term $\gamma^i$ at $V(s_{t+i})$, therefor the approximation of advantage function is less biased .
It, however, is not perfect, since the variance is increasing with longer path, e.g. $\hat{A}^{(1)}_t$ has low variance and high bias, and $\hat{A}^{(\infty)}_t$ has low bias, but high variance. The tradeoff can be introduced by taking some $i < \infty$ and estimating the A by $\hat{A}^{(i)}_t$. But choice of $i$ is not evident. By analogy of generalisation for TD($\lambda$), described in Sutton's book, chapter
12.8 the other way to introduce the trade-off between bias and variance is to take a weighted sum. In practical case, of course the infinite trajectories are not accessible. So the trajectories of length T are taken into account, which results exactly in
$$
\hat{A}_t = \sum_{i=0}^{T-(t+1)} (\gamma \lambda)^i \delta^{V_\theta}_{t + i}
$$
Where $\lambda$ introduces trade off between bias and variance of advantage function estimation.
For further detailed reading I would highly recommend looking at the other article by J. Schulman. It has a section (3. Advantage Function Estimation) on exactly this question.
