3
$\begingroup$

Let's use Excercise 3.8 from Sutton, Barto - Introduction to RL:

Suppose $\gamma = 0.5$ and following sequence of rewards is received $R_1=-1$ , $R_2=2$ , $R_3=6$ , $R_4=3$ , $R_5=2$ , with $T=5$ . What are $G_0, G_1, ..., G_5?$

There isn't $G_5$ because $R_5$ is last reward. Am I understanding it right?

So:

$G_4 = 2$

$G_3 = 3 + 0.5*2 = 4$

$G_2 = 6+0.5*4 = 8$

$G_1 = 2+0.5*8 = 6$

$G_0 = -1 +0.5*6 = 2$

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

Perfect.

To back up your intuition about there not being a $G_5$, refer to the definition of discounted return in the periodic case (3.11). $$G_t \doteq \sum_{k=t+1}^T \gamma^{k-t-1} R_k$$

You'll see that $G_5$ would be written as a sum with no terms in it, since $T=5$.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .