In Barton and Sutton's book, Reinforcement Learning: An Introduction (2nd edition), an expression, on page 289 (equation 12.2), introduced the form of the $\lambda$-return defined as follows

$$G_t^{\lambda} = (1-\lambda)\sum_{n=1}^{\infty} \lambda^{n-1}G_{t:t+n} \label{12.2}\tag{12.2}$$

with the truncated return defined as

$$ G_{t:t+n} \doteq R_{t+1} +\gamma R_{t+2} + \ldots + \gamma^{n-1} R_{t+n} + \gamma^{n}\hat{v}(S_{t+n}, \mathbf{w}_{t+n-1}) \label{12.1}\tag{12.1}$$

However, slightly later in the text, page 290 (equation 12.4), the update algorithm for the offline $\lambda$-return algorithm is defined as

$$ \mathbf{w}_{t+1} \doteq \mathbf{w}_{t}+\alpha\left[G_{t}^{\lambda}-\hat{v}\left(S_{t}, \mathbf{w}_{t}\right)\right] \nabla \hat{v}\left(S_{t}, \mathbf{w}_{t}\right), \quad t=0, \ldots, T-1 \label{12.4}\tag{12.4} $$

My question is: how do we bootstrap the truncated returns in the update algorithm?

The way the truncated return is currently defined can not plausibly be used, since we would not have access to $\mathbf{w}_{t+n-1}$, as we are in the process of finding $\mathbf{w}_{t+1}$. I suspect $\mathbf{w}_{t}$ is used for bootstrapping in all returns, but that would alter the definition of the truncated return which I just wanted to clarify.

And as a follow-up question: What weights are used for bootstrapping in the online $\lambda$-return algorithm described on page 298?

I assume it's either $\mathbf{w}_{t-1}^{h}$ or $\mathbf{w}_{h-1}^{h-1}$, it's briefly mentioned that the online $\lambda$-return algorithm performs slightly better than the offline one at the end of the episode which leads me to believe the latter is used otherwise the two algorithms would be identical.

Any insight into either question would be great.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.