3
$\begingroup$

Sutton and Barto 2018 define the discounted return $G_t$ the following way (p 55):

enter image description here

Is my interpretation correct?

enter image description here

Or should all "1" be in the same column?

$\endgroup$
3
$\begingroup$

Your table is almost correct. It is a minor difference, you should not have a $R_0$, the top row, leftmost column of numbers should be empty. That is because the first reward is $R_1$ (a result of taking action $A_0$ in state $S_0$). The alignment of the columns on the right hand side is correct though.

It might help to add the time step number at the top. But the important detail is that $G_t$ is a measure of all future rewards.

For instance it should always be zero when you reach a terminal state, which is what your example shows. Whilst it is quite common to receive a reward at the end of an episode (i.e. whilst arriving at a terminal state), also as your example shows.

The decision to have reward time step match that of next state is a convention that can be altered. A few RL sources, but not Sutton & Barto, will have reward on same time step as state and action that decided it, and thus $R_0$ will exist. The reward of 1 for reaching the terminal state would then be 1 time step earlier in your table, and there would be no $R_4$. The definition of $G_t$ would need to change to match ($G_t = R_t + \gamma G_{t+1})$, as well as other equations. That would change your table also - the reward sequence (top row) would shift to the left.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.