2
$\begingroup$

Why is it useful to define the return as the sum of the rewards from time $t$ onward rather than up to $t$?

The return for an MDP is usually defined as

$$G_t=R_{t+1}+R_{t+2}+ \dots +R_T$$

Why is this defined as the return? Is there anything useful about this?

It seems like it's more useful to define the return as $$G_t=R_0+ \dots+R_t,$$ because your "return", so to speak, is the "profit from investment" so it seems like your return will be your accumulated reward from taking actions up to that point.

$\endgroup$
0
3
$\begingroup$

It wouldn't make sense to define the return as you propose, from time 0 to $t$. Once we are in a state at time $t$ we don't care what the returns have been, rather what they will be in the future, thus returns are defined as the total sum of discounted rewards from the current time step onwards. This allows the agent to make decisions about which actions to take based on how valuable taking said action is in the current state at time $t$ -- clearly the rewards previous to this have no effect upon that.

$\endgroup$
3
  • $\begingroup$ You say "Once we are in state $t$", but the return does not necessarily depend on any state. I think you're referring to the definition of the state-action (or state) value function, which is the expected return for taking action $a$ in state $s$ and then following a certain policy, so you should clarify that (although the OP accepted this answer, so I guess he's happy with it). $\endgroup$ – nbro Nov 30 '20 at 23:38
  • $\begingroup$ I meant to say a state. Once we are in state $s$ at time $t$ then we are interested in future returns from this time step onwards, regardless of what has happened. $\endgroup$ – David Ireland Dec 1 '20 at 0:52
  • $\begingroup$ Yes, we want to predict the future, but maybe the original doubt arouse because the OP was thinking something like "Shouldn't we also use the past to predict the future?". $\endgroup$ – nbro Dec 1 '20 at 1:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.