Questions tagged [return]

For questions related to the concept of "return" in reinforcement learning.

Filter by
Sorted by
Tagged with
2
votes
1answer
27 views

Why is $G_{t+1}$ is replaced with $v_*(S_{t+1})$ in the Bellman optimality equation?

In equation 3.17 of Sutton and Barto's book: $$q_*(s, a)=\mathbb{E}[R_{t+1} + \gamma v_*(S_{t+1}) \mid S_t = s, A_t = a]$$ $G_{t+1}$ here have been replaced with $v_*(S_{t+1})$, but no reason has ...
2
votes
1answer
30 views

Is there any difference between reward and return in reinforcement learning?

I am reading Sutton and Barto's book on reinforcement learning. I thought that reward and return were the same things. However, in Section 5.6 of the book, 3rd line, first paragraph, it is written: ...
2
votes
1answer
30 views

Shouldn't expected return be calculated for some faraway time in the future $t+n$ instead of current time $t$?

I am learning RL for the first time. It may be naive, but it is a bit odd to grasp this idea that, if the goal of RL is to maximize the expected return, then shouldn't the expected return be ...
2
votes
2answers
57 views

Is my understanding of the value function, Q function, policy, reward and return correct?

I'm a beginner in the RL field, and I would like to check that my understanding of certain RL concepts. Value function: How good it is to be in a state S following policy π. ...
3
votes
2answers
97 views

What is the difference between return and expected return?

At a time step $t$, for a state $S_{t}$, the return is defined as the discounted cumulative reward from that time step $t$. If an agent is following a policy (which in itself is a probability ...
1
vote
1answer
197 views

How can the $\lambda$-return be defined recursively?

The $\lambda$-return is defined as $$G_t^\lambda = (1-\lambda)\sum_{n=1}^\infty \lambda^{n-1}G_{t:t+n}$$ where $$G_{t:t+n} = R_{t+1}+\gamma R_{t+2}+\dots +\gamma^{n-1}R_{t+n} + \gamma^n\hat{v}(S_{t+n})...
3
votes
1answer
57 views

Given specific rewards, how can I calculate the returns for each time step?

Let's use Excercise 3.8 from Sutton, Barto - Introduction to RL: Suppose $\gamma = 0.5$ and following sequence of rewards is received $R_1=-1$ , $R_2=2$ , $R_3=6$ , $R_4=3$ , $R_5=2$ , with $T=5$ ...
5
votes
2answers
439 views

Why are lambda returns so rarely used in policy gradients?

I've seen monte-carlo reward $G_{t}$ used in REINFORCE and TD($0$) reward $r_t + \gamma Q(s', a')$ used in vanilla actor-critic. I've never seen someone use lambda reward $G^{\lambda}_{t}$ in these ...
3
votes
1answer
63 views

Is my interpretation of the return correct?

Sutton and Barto 2018 define the discounted return $G_t$ the following way (p 55): Is my interpretation correct? Or should all "1" be in the same column?