Sutton-Barto, second edt, page 128, Exercise 6.7:

Design an off-policy version of the TD(0) update that can be used with arbitrary target policy $\pi$ and covering behavior policy $b$, using at each step t the importance sampling ratio $\rho_{t:t}$ (5.3).

The answer is:

$V(S_t) \leftarrow V(S_t) + \alpha\rho_{t:t}\left[R_{t+1} + V(S_{t+1})-V(S_t)\right]$

Normally, if we make a similarity to Monte Carlo method, the target here is $R_{t+1} + V(S_{t+1})$ and hence I was expecting the answer as:

$V(S_t) \leftarrow V(S_t) + \alpha\left[\rho_{t:t}(R_{t+1} + V(S_{t+1}))-V(S_t)\right]$

Where is my mistake? Or, is the answer wrong?

  • $\begingroup$ Where are you getting the official answer from? They are not published in the book. $\endgroup$ Apr 21 at 8:32
  • $\begingroup$ I have the solutions manual. $\endgroup$ Apr 21 at 21:12
  • $\begingroup$ There is more than one place with published solutions. You can even get them from Sutton if you contact him and ask. Please give more information, link if online. $\endgroup$ Apr 22 at 6:24
  • 1
    $\begingroup$ Please, put your specific question in the title. "off-policy TD(0)" is far from being a good title because it could be about 100 things. $\endgroup$
    – nbro
    Apr 22 at 13:32


You must log in to answer this question.

Browse other questions tagged .