How does n-step TD removes the notion of time-step as referenced in Sutton and Barto (2nd edition, Page 163) below?

Another way of looking at the benefits of n-step methods is that they free you from the tyranny of the time step. With one-step TD methods the same time step determines how often the action can be changed and the time interval over which bootstrapping is done. With one-step TD methods, these time intervals are the same, and so a compromise must be made. n-step methods enable bootstrapping to occur over multiple steps, freeing us from the tyranny of the single time step.

Consider the following example.

We know our n-step update equation as: $$V_{t+n}(S_t) = V_{t+n-1}(S_t) + \alpha [G_{t:t+n} - V_{t+n-1}(S_t)]$$

Now, let $t=0$ and $n=2$. This gives us: $V_2(S_0) = V_1(S_0) + \alpha [G_{0:2} - V_1(S_0)]$.

Before our n-step TD prediction algorithm starts, we initialize with $V_0$. But we use $V_1$. Why? And how do we calculate $V_1$?

  • $\begingroup$ It seems to me you're asking 2 distinct questions here. "how do we calculate V1?" and "how does n-step TD removes the notion of time-step?". Please, edit your post to leave just one question per post, although the questions are related to the same topic. Ask the second question in a different post. $\endgroup$
    – nbro
    Aug 21 at 10:41
  • $\begingroup$ Actually, I wanted to use the former question as an example to illustrate actual question I would like to be answered. I edited the question again. If the issue still persists, I'll split the questions into different post. $\endgroup$
    – user529295
    Aug 21 at 17:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.