2
$\begingroup$

Why is it useful to define the return as the sum of the rewards from time $t$ onward rather than up to $t$?

The return for an MDP is usually defined as

$$G_t=R_{t+1}+R_{t+2}+ \dots +R_T$$

Why is this defined as the return? Is there anything useful about this?

It seems like it's more useful to define the return as $$G_t=R_0+ \dots+R_t,$$ because your "return", so to speak, is the "profit from investment" so it seems like your return will be your accumulated reward from taking actions up to that point.

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
3
$\begingroup$

It wouldn't make sense to define the return as you propose, from time 0 to $t$. Once we are in a state at time $t$ we don't care what the returns have been, rather what they will be in the future, thus returns are defined as the total sum of discounted rewards from the current time step onwards. This allows the agent to make decisions about which actions to take based on how valuable taking said action is in the current state at time $t$ -- clearly the rewards previous to this have no effect upon that.

Improve this answer
$\endgroup$
3
  • $\begingroup$ You say "Once we are in state $t$", but the return does not necessarily depend on any state. I think you're referring to the definition of the state-action (or state) value function, which is the expected return for taking action $a$ in state $s$ and then following a certain policy, so you should clarify that (although the OP accepted this answer, so I guess he's happy with it). $\endgroup$
    – nbro
    Nov 30, 2020 at 23:38
  • $\begingroup$ I meant to say a state. Once we are in state $s$ at time $t$ then we are interested in future returns from this time step onwards, regardless of what has happened. $\endgroup$
    – David
    Dec 1, 2020 at 0:52
  • $\begingroup$ Yes, we want to predict the future, but maybe the original doubt arouse because the OP was thinking something like "Shouldn't we also use the past to predict the future?". $\endgroup$
    – nbro
    Dec 1, 2020 at 1:11

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .