# Questions tagged [return]

For questions related to the concept of return in reinforcement learning, which is defined as the future cumulative (discounted) reward or, in simple words, the reward in the long run.

15 questions
Filter by
Sorted by
Tagged with
79 views

### Is the expected value we sample in TD-learning action-value Q or state-value V?

Both MC and TD are model-free and they both follow a sample trajectory (in the case of TD, the trajectory is cut-short) to estimate the return (we basically are sampling Q values). Other than that, ...
53 views

### When learning off-policy with multi-step returns, why do we use the current behaviour policy in importance sampling?

When learning off-policy with multi-step returns, we want to update the value of $Q(s_1, a_1)$ using rewards from the trajectory $\tau = (s_1, a_1, r_1, s_2, a_2, r_2, ..., s_n, a_n, r_n, s_n+1)$. We ...
88 views

75 views

### Given specific rewards, how can I calculate the returns for each time step?

Let's use Excercise 3.8 from Sutton, Barto - Introduction to RL: Suppose $\gamma = 0.5$ and following sequence of rewards is received $R_1=-1$ , $R_2=2$ , $R_3=6$ , $R_4=3$ , $R_5=2$ , with $T=5$ ...
603 views

### Why are lambda returns so rarely used in policy gradients?

I've seen monte-carlo reward $G_{t}$ used in REINFORCE and TD($0$) reward $r_t + \gamma Q(s', a')$ used in vanilla actor-critic. I've never seen someone use lambda reward $G^{\lambda}_{t}$ in these ...
69 views

### Is my interpretation of the return correct?

Sutton and Barto 2018 define the discounted return $G_t$ the following way (p 55): Is my interpretation correct? Or should all "1" be in the same column?