Sutton-Barto, page 164:

enter image description here

In the (f) loop, $S,A, S', R$ are from real experience (Model(S,A)=(R, S') where R and S' are also real expereince). This experience is used in direct RL part in (d). Why is the same experience used again in the (f) part? Q-update in (d) uses the same info as Q-update in (f). I am totally confused.

It seems to me that part (f) is a n-times re-application in parts before it.


1 Answer 1


The hope in (f) is that the ${max}_aQ(S^\prime,a)$ has changed since it was previously evaluated so that it's effect on the present state's value can be propagated without the agent having to spend the steps to experience it.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .