3
$\begingroup$

I am trying to understand the algorithm for n-step Sarsa from Sutton/Barto (2nd Edition, p. 157, PDF) As I understand it, this algorithm should update n state action values, but I cannot see where it is 'propagated backwards' (sorry for the wrong terminology, but I couldn't find something better). Probably, I am not seeing the forrest for all the trees?

$\endgroup$
1
$\begingroup$

The important part, where you can see a single reward value is used for $n$ different updates, is the part where a sum of $R_i$ values with $i$ ranging from $\tau + 1$ to $\tau + n$ is assigned to $G$.

So yes, the outer loop of the algorithm always does at most one update per iteration, but for that update it uses multiple previously observed $R_i$ values. Each of those $R_i$ values is used for multiple updates (not multiple updates at the same time, but multiple updates spread out over different iterations).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.