In reinforcement learning, the return is defined as some function of the rewards. For example, you can have the discounted return, where you multiply the rewards received at later time steps by increasingly smaller numbers, so that the rewards closer to the current time step have a higher weight. You can also have $n$-step returns or $\lambda$-returns.

Recently, I have come across the concept of return-to-go in a few research papers, such as Prioritized Experience Replay (appendix A. Prioritization Variants, p. 12) or Being Optimistic to Be Conservative: Quickly Learning a CVaR Policy (section Theoretical Analysis, p. 3).

What exactly is the return-to-go? How is it mathematically defined? In which situations do we need to care about it? The name suggests that this is the return starting from a certain time step $t$, but wouldn't this be the same thing as the return (which is defined starting from a certain time step $t$ and often denoted as $G_t$ for that same reason)?

There is also the concept of reward-to-go. For example, the reward-to-go is analyzed in the paper Learning the Variance of the Reward-To-Go, which states that the expected reward-to-go is the value function, which seems to be consistent with this explanation of the reward-to-go, where the reward-to-go is defined as

$$\hat{R}_{t} \doteq \sum_{t^{\prime}=t}^{T} R\left(s_{t^{\prime}}, a_{t^{\prime}}, s_{t^{\prime}+1}\right)$$

We also had a few questions that involve the reward-to-go: for example, this or this. How is the return-to-go related to the reward-to-go? Are they the same thing? For example, in this paper, the return-to-go seems to be used as a synonym for reward-to-go (as used in this article), i.e. they call $R(t)$ the "return to-go" (e.g. on page 2), which should be the return starting from time step $t$, which should actually be the reward-to-go.

  • $\begingroup$ "return-to-go" is probably not standard terminology, but I think that some people will also have this question (if they come across the same papers), that's why I asked it (i.e. to serve as a future reference). $\endgroup$
    – nbro
    Oct 10 '20 at 16:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.