Questions tagged [return]

For questions related to the concept of "return" in reinforcement learning.

Filter by
Sorted by
Tagged with
2
votes
1answer
84 views

Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards?

Why is the expected return in Reinforcement Learning (RL) computed as a sum of cumulative rewards? Would it not make more sense to compute $\mathbb{E}(R \mid s, a)$ (the expected return for taking ...
1
vote
2answers
53 views

How do I calculate the return given the discount factor and a sequence of rewards?

I know that $G_t = R_{t+1} + G_{t+1}$. Suppose $\gamma = 0.9$ and the reward sequence is $R_1 = 2$ followed by an infinite sequence of $7$s. What is the value of $G_0$? As it's infinite, how can we ...
1
vote
0answers
40 views

Why does the n-step return being zero result in high variance in off policy n-step TD?

In the paragraph given between eq 7.12 and 7.13 in Sutton & Barto's book: $G_{t:h} = R_{t+1} + G_{t+1:h} , t < h < T$ where $G_{h:h} = V_{h-1}(S_h)$. (Recall that this return is used at ...
4
votes
2answers
98 views

Why is $G_{t+1}$ is replaced with $v_*(S_{t+1})$ in the Bellman optimality equation?

In equation 3.17 of Sutton and Barto's book: $$q_*(s, a)=\mathbb{E}[R_{t+1} + \gamma v_*(S_{t+1}) \mid S_t = s, A_t = a]$$ $G_{t+1}$ here have been replaced with $v_*(S_{t+1})$, but no reason has ...
3
votes
2answers
68 views

Is there any difference between reward and return in reinforcement learning?

I am reading Sutton and Barto's book on reinforcement learning. I thought that reward and return were the same things. However, in Section 5.6 of the book, 3rd line, first paragraph, it is written: ...
2
votes
1answer
38 views

Shouldn't expected return be calculated for some faraway time in the future $t+n$ instead of current time $t$?

I am learning RL for the first time. It may be naive, but it is a bit odd to grasp this idea that, if the goal of RL is to maximize the expected return, then shouldn't the expected return be ...
2
votes
2answers
75 views

Is my understanding of the value function, Q function, policy, reward and return correct?

I'm a beginner in the RL field, and I would like to check that my understanding of certain RL concepts. Value function: How good it is to be in a state S following policy π. ...
3
votes
2answers
111 views

What is the difference between return and expected return?

At a time step $t$, for a state $S_{t}$, the return is defined as the discounted cumulative reward from that time step $t$. If an agent is following a policy (which in itself is a probability ...
1
vote
1answer
243 views

How can the $\lambda$-return be defined recursively?

The $\lambda$-return is defined as $$G_t^\lambda = (1-\lambda)\sum_{n=1}^\infty \lambda^{n-1}G_{t:t+n}$$ where $$G_{t:t+n} = R_{t+1}+\gamma R_{t+2}+\dots +\gamma^{n-1}R_{t+n} + \gamma^n\hat{v}(S_{t+n})...
3
votes
1answer
70 views

Given specific rewards, how can I calculate the returns for each time step?

Let's use Excercise 3.8 from Sutton, Barto - Introduction to RL: Suppose $\gamma = 0.5$ and following sequence of rewards is received $R_1=-1$ , $R_2=2$ , $R_3=6$ , $R_4=3$ , $R_5=2$ , with $T=5$ ...
5
votes
2answers
502 views

Why are lambda returns so rarely used in policy gradients?

I've seen monte-carlo reward $G_{t}$ used in REINFORCE and TD($0$) reward $r_t + \gamma Q(s', a')$ used in vanilla actor-critic. I've never seen someone use lambda reward $G^{\lambda}_{t}$ in these ...
3
votes
1answer
67 views

Is my interpretation of the return correct?

Sutton and Barto 2018 define the discounted return $G_t$ the following way (p 55): Is my interpretation correct? Or should all "1" be in the same column?