# Is the TD-residual defined for timesteps $t$ past the length of the episode?

Let $$\mathcal{S}$$ be the state-space in a reinforcement learning problem where rewards are in $$\mathbb{R}$$, and let $$V:\mathcal{S} \to \mathbb{R}$$ be an approximate value function. Following the GAE paper, the TD-residual with discount $$\gamma \in [0,1]$$ is defined as $$\delta_t^V = r_t + \gamma V(s_{t + 1}) - V(s_t)$$.

I am confused by the formula for the GAE-$$\lambda$$ advantage estimator, which is $$\hat{A}_t^{\text{GAE}(\gamma, \lambda)} = \sum_{l = 0}^\infty (\gamma \lambda)^l \delta_{t + l}^V.$$

This seems to imply that $$\delta_t^V$$ is defined for $$t > N$$, where $$N$$ is the length of the current trajectory/episode. It looks like in implementations of this advantage estimator, it is just assumed that $$\delta_t^V = 0$$ for $$t > N$$, since the sums are finite. Is there justification for this assumption? Or am I missing something here? All help appreciated, thanks.

Edit: a link to the GAE paper: https://arxiv.org/abs/1506.02438

• Maybe you could also provide a link to the GAE paper. – nbro Apr 3 at 18:09