# Is my interpretation of the return correct?

Sutton and Barto 2018 define the discounted return $$G_t$$ the following way (p 55): Is my interpretation correct? Or should all "1" be in the same column?

## 1 Answer

Your table is almost correct. It is a minor difference, you should not have a $R_0$, the top row, leftmost column of numbers should be empty. That is because the first reward is $R_1$ (a result of taking action $A_0$ in state $S_0$). The alignment of the columns on the right hand side is correct though.

It might help to add the time step number at the top. But the important detail is that $G_t$ is a measure of all future rewards.

For instance it should always be zero when you reach a terminal state, which is what your example shows. Whilst it is quite common to receive a reward at the end of an episode (i.e. whilst arriving at a terminal state), also as your example shows.

The decision to have reward time step match that of next state is a convention that can be altered. A few RL sources, but not Sutton & Barto, will have reward on same time step as state and action that decided it, and thus $R_0$ will exist. The reward of 1 for reaching the terminal state would then be 1 time step earlier in your table, and there would be no $R_4$. The definition of $G_t$ would need to change to match ($G_t = R_t + \gamma G_{t+1})$, as well as other equations. That would change your table also - the reward sequence (top row) would shift to the left.