0
$\begingroup$

Sutton-Barto, page 164:

enter image description here

In the (f) loop, $S,A, S', R$ are from real experience (Model(S,A)=(R, S') where R and S' are also real expereince). This experience is used in direct RL part in (d). Why is the same experience used again in the (f) part? Q-update in (d) uses the same info as Q-update in (f). I am totally confused.

It seems to me that part (f) is a n-times re-application in parts before it.

$\endgroup$

1 Answer 1

3
$\begingroup$

The hope in (f) is that the ${max}_aQ(S^\prime,a)$ has changed since it was previously evaluated so that it's effect on the present state's value can be propagated without the agent having to spend the steps to experience it.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .