# Questions tagged [return]

For questions related to the concept of return in reinforcement learning, which is defined as the future cumulative (discounted) reward or, in simple words, the reward in the long run.

15 questions
Filter by
Sorted by
Tagged with
79 views

### Is the expected value we sample in TD-learning action-value Q or state-value V?

Both MC and TD are model-free and they both follow a sample trajectory (in the case of TD, the trajectory is cut-short) to estimate the return (we basically are sampling Q values). Other than that, ...
53 views

### When learning off-policy with multi-step returns, why do we use the current behaviour policy in importance sampling?

When learning off-policy with multi-step returns, we want to update the value of $Q(s_1, a_1)$ using rewards from the trajectory $\tau = (s_1, a_1, r_1, s_2, a_2, r_2, ..., s_n, a_n, r_n, s_n+1)$. We ...
132 views

### What is the difference between return and expected return?

At a time step $t$, for a state $S_{t}$, the return is defined as the discounted cumulative reward from that time step $t$. If an agent is following a policy (which in itself is a probability ...
88 views

Equation 7.3 of Sutton Barto book: $$\text{Equation: } max_s|\mathbb{E}_\pi[G_{t:t+n}|S_t = s] - v_\pi| \le \gamma^nmax_s|V_{t+n-1}(s) - v_\pi(s)|$$ $$\text{where }G_{t:t+n} = R_{t+1} + \gamma R_{t+2}... 2answers 603 views ### Why are lambda returns so rarely used in policy gradients? I've seen monte-carlo reward G_{t} used in REINFORCE and TD(0) reward r_t + \gamma Q(s', a') used in vanilla actor-critic. I've never seen someone use lambda reward G^{\lambda}_{t} in these ... 1answer 126 views ### Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards? Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards? Would it not make more sense to compute \mathbb{E}(R \mid s, a) (the expected return for taking ... 2answers 58 views ### How do I calculate the return given the discount factor and a sequence of rewards? I know that G_t = R_{t+1} + G_{t+1}. Suppose \gamma = 0.9 and the reward sequence is R_1 = 2 followed by an infinite sequence of 7s. What is the value of G_0? As it's infinite, how can we ... 0answers 48 views ### Why does the n-step return being zero result in high variance in off policy n-step TD? In the paragraph given between eq 7.12 and 7.13 in Sutton & Barto's book: G_{t:h} = R_{t+1} + G_{t+1:h} , t < h < T where G_{h:h} = V_{h-1}(S_h). (Recall that this return is used at ... 2answers 87 views ### Is there any difference between reward and return in reinforcement learning? I am reading Sutton and Barto's book on reinforcement learning. I thought that reward and return were the same things. However, in Section 5.6 of the book, 3rd line, first paragraph, it is written: ... 2answers 107 views ### Why is G_{t+1} is replaced with v_*(S_{t+1}) in the Bellman optimality equation? In equation 3.17 of Sutton and Barto's book:$$q_*(s, a)=\mathbb{E}[R_{t+1} + \gamma v_*(S_{t+1}) \mid S_t = s, A_t = a]$$G_{t+1} here have been replaced with v_*(S_{t+1}), but no reason has ... 1answer 39 views ### Shouldn't expected return be calculated for some faraway time in the future t+n instead of current time t? I am learning RL for the first time. It may be naive, but it is a bit odd to grasp this idea that, if the goal of RL is to maximize the expected return, then shouldn't the expected return be ... 2answers 86 views ### Is my understanding of the value function, Q function, policy, reward and return correct? I'm a beginner in the RL field, and I would like to check that my understanding of certain RL concepts. Value function: How good it is to be in a state S following policy Ļ. ... 1answer 305 views ### How can the \lambda-return be defined recursively? The \lambda-return is defined as$$G_t^\lambda = (1-\lambda)\sum_{n=1}^\infty \lambda^{n-1}G_{t:t+n}$$where$$G_{t:t+n} = R_{t+1}+\gamma R_{t+2}+\dots +\gamma^{n-1}R_{t+n} + \gamma^n\hat{v}(S_{t+n})...
Let's use Excercise 3.8 from Sutton, Barto - Introduction to RL: Suppose $\gamma = 0.5$ and following sequence of rewards is received $R_1=-1$ , $R_2=2$ , $R_3=6$ , $R_4=3$ , $R_5=2$ , with $T=5$ ...
Sutton and Barto 2018 define the discounted return $G_t$ the following way (p 55): Is my interpretation correct? Or should all "1" be in the same column?