Let $\mathcal{S}$ be the state-space in a reinforcement learning problem where rewards are in $\mathbb{R}$, and let $V:\mathcal{S} \to \mathbb{R}$ be an approximate value function. Following the GAE paper, the TD-residual with discount $\gamma \in [0,1]$ is defined as $\delta_t^V = r_t + \gamma V(s_{t + 1}) - V(s_t)$.

I am confused by the formula for the GAE-$\lambda$ advantage estimator, which is $$ \hat{A}_t^{\text{GAE}(\gamma, \lambda)} = \sum_{l = 0}^\infty (\gamma \lambda)^l \delta_{t + l}^V. $$

This seems to imply that $\delta_t^V$ is defined for $t > N$, where $N$ is the length of the current trajectory/episode. It looks like in implementations of this advantage estimator, it is just assumed that $\delta_t^V = 0$ for $t > N$, since the sums are finite. Is there justification for this assumption? Or am I missing something here? All help appreciated, thanks.

Edit: a link to the GAE paper: https://arxiv.org/abs/1506.02438

  • $\begingroup$ Maybe you could also provide a link to the GAE paper. $\endgroup$ – nbro Apr 3 at 18:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.