3
$\begingroup$

Let's use Excercise 3.8 from Sutton, Barto - Introduction to RL:

Suppose $\gamma = 0.5$ and following sequence of rewards is received $R_1=-1$ , $R_2=2$ , $R_3=6$ , $R_4=3$ , $R_5=2$ , with $T=5$ . What are $G_0, G_1, ..., G_5?$

There isn't $G_5$ because $R_5$ is last reward. Am I understanding it right?

So:

$G_4 = 2$

$G_3 = 3 + 0.5*2 = 4$

$G_2 = 6+0.5*4 = 8$

$G_1 = 2+0.5*8 = 6$

$G_0 = -1 +0.5*6 = 2$

$\endgroup$
1
$\begingroup$

Perfect.

To back up your intuition about there not being a $G_5$, refer to the definition of discounted return in the periodic case (3.11). $$G_t \doteq \sum_{k=t+1}^T \gamma^{k-t-1} R_k$$

You'll see that $G_5$ would be written as a sum with no terms in it, since $T=5$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.