6
$\begingroup$

In reinforcement learning, the return is defined as some function of the rewards. For example, you can have the discounted return, where you multiply the rewards received at later time steps by increasingly smaller numbers, so that the rewards closer to the current time step have a higher weight. You can also have $n$-step returns or $\lambda$-returns.

Recently, I have come across the concept of return-to-go in a few research papers, such as Prioritized Experience Replay (appendix A. Prioritization Variants, p. 12) or Being Optimistic to Be Conservative: Quickly Learning a CVaR Policy (section Theoretical Analysis, p. 3).

What exactly is the return-to-go? How is it mathematically defined? In which situations do we need to care about it? The name suggests that this is the return starting from a certain time step $t$, but wouldn't this be the same thing as the return (which is defined starting from a certain time step $t$ and often denoted as $G_t$ for that same reason)?

There is also the concept of reward-to-go. For example, the reward-to-go is analyzed in the paper Learning the Variance of the Reward-To-Go, which states that the expected reward-to-go is the value function, which seems to be consistent with this explanation of the reward-to-go, where the reward-to-go is defined as

$$\hat{R}_{t} \doteq \sum_{t^{\prime}=t}^{T} R\left(s_{t^{\prime}}, a_{t^{\prime}}, s_{t^{\prime}+1}\right)$$

We also had a few questions that involve the reward-to-go: for example, this or this. How is the return-to-go related to the reward-to-go? Are they the same thing? For example, in this paper, the return-to-go seems to be used as a synonym for reward-to-go (as used in this article), i.e. they call $R(t)$ the "return to-go" (e.g. on page 2), which should be the return starting from time step $t$, which should actually be the reward-to-go.

$\endgroup$
2
  • 1
    $\begingroup$ "return-to-go" is probably not standard terminology, but I think that some people will also have this question (if they come across the same papers), that's why I asked it (i.e. to serve as a future reference). $\endgroup$
    – nbro
    Oct 10, 2020 at 16:00
  • $\begingroup$ so have you figured it out? $\endgroup$
    – Sam
    Jan 27, 2023 at 17:18

1 Answer 1

1
$\begingroup$

Short answer: I the return-to-go is just a synonym of the reward-to-go, which instead is a well known term used to define the basic form of policy gradient, e.g. REINFORCE-like.

Reward-to-go

As you correctly wrote, the reward-to-go is just a sum of rewards collected in a trajectory $\tau$ from a certain timestep $t$ until termination $T$. If we define $R(\tau)=\sum_{t=1}^T r_t^{\tau}$ as the sum of immediate rewards $r_t$ got in trajectory $\tau$, from that we can define the rewards-to-go just by considering a subset of the trajectory, so discarding the past, i.e starting from a time $t$: $$R_t(\tau)=\sum_{t'=t}r_{t'}^\tau$$

In particular, the quantity $R_t(\tau)$ figures in the (basic) policy gradient formulation: $$\nabla_\theta J(\theta)\approx \sum_i\Big(\sum_{t=1}^T \nabla_\theta\log \pi_\theta\big(a_t^{(i)}\mid s_t^{(i)}\big) R_t\big(\tau^{(i)}\big) \Big)$$

Basically, the rewards-to-go score actions within a trajectory $\tau^{(i)}$: the spinning-up rl article you mentioned refers to the fact that the original (although naive) PG formulation uses $R(\tau)$, i.e. sum of rewards, to score the policy's actions but it's useless to consider the immediate rewards $r_k$ gained in the past (so when $k<t$) to score actions happened at time $t$ and onward until the end of the trajectory $T$. In this regard, both $R(\tau)$ and $R_t(\tau)$ are empirical estimates of the value of an action $a_t$ from a given state $s_t$ - think of it as a single-sample estimate of the action-value function, $Q(s_t,a_t)\approx R_t$ - but $R(\tau)$ has a higher variance due the additional sums.

This is reasonable, since in more recent PG formulations we subtract a baseline function $b(s)$ to the rewards-to-go to reduce the variance of the gradient estimate: $\nabla_\theta\log\pi_\theta(a_t\mid s_t)(R_t - b(s_t))$. If $b(s)=V(s)$ is chosen to be the value function, then $R_t - b(s_t)$ is an estimate of the advantage function $A(s_t, a_t) = Q(s_t,a_t) - V(s_t)$ since $R_t$ is the action-value gathered in the current trajectory (but not expected, as in the Q-function.) Formulations that use $A(s,a)$ yield the gradient with lowest variance: A2C uses this.

In which situations do we need to care about it?

We talk about rewards-to-go mainly when dealing with the PG estimation, and defining new baseline functions to improve the gradient estimate: otherwise the term return is largely preferred. Note: it is demonstrated that subtracting a baseline does not alter the gradient formulation, and if correctly picked it does not either add bias to the estimate.

the reward-to-go is analyzed in the paper which states that the expected reward-to-go is the value function

The rewards-to-go are more appropriate as action-value estimates rather than state-value estimates (i.e. the return $G_t$) since we start from a state-action pair. This is a technicality that does not hold in practice, since the value function is also estimated with empirical returns (i.e. the return of a single trajectory $\tau$), i.e. $G_t(\tau) = r_t^\tau + r_{t+1}^\tau +\cdots + r_T^\tau$ that correspond to the rewards-to-go $R_t(\tau)$: think about learning a value function in deep RL, we just fit the neural-net to empirical returns than MSE + GD does the magic. Now, if we average the collected rewards or returns over all possible trajectories we obtain the expected return that is the value function: $V(s_t) = \mathbb{E}_\tau[G_t(\tau)] = \mathbb{E}_\tau[R_t(\tau)] = \mathbb{E}_\tau[r_t^\tau + r_{t+1}^\tau + \cdots + r_{T}^\tau\mid s_t]$.

How is the return-to-go related to the reward-to-go?

They are same thing. I'd say to avoid "return-to-go" and use return, instead (or at most reward-to-go when talking about PG.)


As a side note: none of both terms are mentioned in the famous book "An introduction to reinforcement learning" by Sutton and Barto, so I guess it's something introduced in modern deep RL.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .