1
$\begingroup$

This is n-step TD(0) update rule:

$v_{k+n}(x_k)=v_{k+n-1}(x_k)+\alpha [g_{k:k+n}-v_{k+n-1}(x_k)]$

Why is the subscript on the left hand-side of equation "k+n", not "k+n-1"? Does the subscript for $v$ denote time step or iteration number? I am confused.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

n-step TD(0) is an extension of TD(0) that uses a sequence of forward n rewards and estimated state values to update the value estimate of the current state $x_k$, and TD(0) and Monte Carlo prediction are two extreme cases of it. In some reference such as Sutton & Barto’s the subscript is using $t$ instead of your $k$ and you can treat it simply as time step of the MDP though nothing prevents you from treating it as iteration number within a loop of an episode. And finally the stochastic approximation iterative equation above is probably not real code implementation thus it has to keep exact time step (k+n) on the LHS.

Improve this answer
$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .