Let $\mathcal{S}$ be the state-space in a reinforcement learning problem where rewards are in $\mathbb{R}$, and let $V:\mathcal{S} \to \mathbb{R}$ be an approximate value function. Following the GAE paper, the TD-residual with discount $\gamma \in [0,1]$ is defined as $\delta_t^V = r_t + \gamma V(s_{t + 1}) - V(s_t)$.

I am confused by the formula for the GAE-$\lambda$ advantage estimator, which is $$ \hat{A}_t^{\text{GAE}(\gamma, \lambda)} = \sum_{l = 0}^\infty (\gamma \lambda)^l \delta_{t + l}^V. $$

This seems to imply that $\delta_t^V$ is defined for $t > N$, where $N$ is the length of the current trajectory/episode. It looks like in implementations of this advantage estimator, it is just assumed that $\delta_t^V = 0$ for $t > N$, since the sums are finite. Is there a justification for this assumption? Or am I missing something here?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.