3
$\begingroup$

I am trying to understand the algorithm for n-step SARSA from Sutton and Barto (2nd Edition). As I understand it, this algorithm should update n state-action values, but I cannot see where it is propagated backward. Can someone explain to me how that works?

$\endgroup$
1
$\begingroup$

The important part, where you can see a single reward value is used for $n$ different updates, is the part where a sum of $R_i$ values with $i$ ranging from $\tau + 1$ to $\tau + n$ is assigned to $G$.

So yes, the outer loop of the algorithm always does at most one update per iteration, but for that update it uses multiple previously observed $R_i$ values. Each of those $R_i$ values is used for multiple updates (not multiple updates at the same time, but multiple updates spread out over different iterations).

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.