Below is the Exercise 6.12 from Sutto-Barto and its solution (from the solution manual) but I was not able to understand it. I will be happy if one can make it clearer.

enter image description here

  • $\begingroup$ What haven't you understood? Can you be more specific in your questions? Your posts look so lazy and it seems you don't care to even specify what you don't understand. $\endgroup$
    – nbro
    Commented Apr 26 at 12:57

1 Answer 1


SARSA requires a tuple $S,A,R,S',A'$ to do an update, where $A'$ is the action you have taken at state $S'$, which means that you can only do the update once you are at state $S''$, where instead Q-learning can do the update at step $S'$

Indeed, you can see that is a subtle difference in the implementation of the methods, however, for the sake of the exercise, that's the reason

  • $\begingroup$ The answer is still not clear and short. Could you try to make it better? $\endgroup$ Commented Apr 25 at 18:55
  • $\begingroup$ Note that SARSA actually does not need to have already reached $S''$ before it can do an update for $Q(S, A)$. Just like Q-learning, it can already run its update after reaching $S'$. However, it does need to have already determined which action it wants to execute in $S'$. But it doesn't have to have already executed that action, only determined what it will be. $\endgroup$
    – Dennis Soemers
    Commented Apr 25 at 19:40
  • 1
    $\begingroup$ @DennisSoemers yes, what I meant as you are saying in a much better way, is that it need to first decide the action, then update, thus if the update changes the greedy action (as in the example), they differ in what will happen $\endgroup$
    – Alberto
    Commented Apr 25 at 23:20
  • $\begingroup$ It "might* help the OP if you showed the difference in the inner loop as pseudocode with the point at which action taken for the loop highlighted. Most notably SARSA will pass the action choice forward to the next iteration where Q learning doesn't need to (although it could so the S&B question is referring to a specific implementation detail) $\endgroup$ Commented Apr 26 at 7:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .