1
$\begingroup$

I am having some trouble with answering the following question:

A rat is involved in an experiment. It experiences one episode. At the first step it hears a bell. At the second step it sees a light. At the third step it both hears a bell and sees a light. It then receives some food, worth +1 reward, and the episode terminates on the fourth step. All other rewards were zero. The experiment is undiscounted ($\gamma = 1$). Approximate the state-value function by a linear combination of these features with two parameters: $b · bell(s) + l · light(s)$. If $b = 2$ and $l = −2$, write down the sequence of $\lambda$-returns $v_t^\lambda$ corresponding to this episode, for $\lambda = 0.5$.

The provided answer is this:

$v^\lambda_1 = 0.5 (−2 + 0.5×0 + 0.5×1) = −3/4$

$v^\lambda_2 = 0.5 (0 + 1×1) = 1/2$

$v^\lambda_3 = 0.5 (2×1) = 1$

However, these are my calculations:

$v^\lambda_1 = 0.5 (.5^0×−2 + .5^1×0 + .5^2×1) = 0.875$

$v^\lambda_2 = 0.5 (.5^0×0 + .5^1×1) = 1/4$

$v^\lambda_3 = 0.5 (.5^0×1) = 1/2$

Any help is appreciated!

Edit: These are the functions used:

$v^{(n)}_t = r_{t+1} +\gamma r_{t+2}+\dots+\gamma^{n−1} r_{t+n} + \gamma^n V_{(s_{t+n})} $

$v^\lambda _t = (1−\lambda)\sum_{n=1}^\infty \lambda^{n−1} v^{(n)}_t$

bell state = $[1 \ \ 0]^T$

light state = $[0 \ \ 1]^T$

bell and light state = $[1 \ \ 1]^T$

$\endgroup$
6
  • 1
    $\begingroup$ Instead of writing "TD ... question", can you please just ask your specific question in the title? Thanks $\endgroup$
    – nbro
    Commented Apr 17 at 0:41
  • $\begingroup$ Could you clarify what formula you are applying in your calculations please, by adding its algebraic form? Most likely you have the wrong formula for the TD plus linear state value approximation, but you could also be making incorrect substitutions. $\endgroup$ Commented Apr 18 at 17:23
  • $\begingroup$ @NeilSlater I've added the formulas $\endgroup$
    – Nat
    Commented Apr 20 at 10:12
  • $\begingroup$ Thanks. I don't see the linear approximator in your formulae? Typically the trace works directly on the approximator's state description vector. $\endgroup$ Commented Apr 20 at 12:24
  • $\begingroup$ @NeilSlater Thanks, I've added it $\endgroup$
    – Nat
    Commented Apr 21 at 18:57

0

You must log in to answer this question.

Browse other questions tagged .