SARSA is on-policy, while n-step SARSA is off-policy. But when n = 1, is it like an off-policy version of SARSA? Any similarity and difference between 1-step SARSA and SARSA?


N-step SARSA can be both off policy and on-policy. I think you already know the n step on-policy SARSA. So I am just telling you how n-step SARSA can be off-policy.

Off-policy n-step SARSA: Now you have two policies, one is target policy, $\pi$, (let's say it is a greedy policy), another one is behavior policy, $b$, (you are actually following this behavior policy). Since this is off-policy, you do importance sampling for that. So the update rule is like this:

$$Q_{t+n}(S_{t},A_{t}) = Q_{t+n-1}(S_{t},A_{t}) + \alpha \rho_{t+1:t+n-1}[G_{t:t+n} - Q_{t+n-1}(S_{t},A_{t})],$$


$$\rho_{t:h} = \prod_{t=k}^{h} \frac{\pi(A_{k}|S_{k})}{b(A_{k}|S_{k})}$$

You are following the behavior policy $b$, but shifting the Q values towards the target policy, $\pi$.

Off-policy one-step SARSA: You can think of Q learning as one-step off policy SARSA.

  • $\begingroup$ Thanks for your reply. Does that mean on-policy 1-step SARSA is the same as SARSA? $\endgroup$
    – Tracy Yang
    Apr 27 '20 at 14:32
  • $\begingroup$ Yes, one-step on-policy SARSA is called SARSA $\endgroup$ Apr 27 '20 at 15:13
  • $\begingroup$ Then what is the difference between Q-learning and 1-step off policy SARSA? I understand the algorithms are different. 1-step off policy SARSA uses importance sampling, and Q learning seems just following behavior policy. What would be different results applying these two algorithms? Are they equivalent theoretically? $\endgroup$
    – Tracy Yang
    Apr 29 '20 at 21:11
  • $\begingroup$ Q learning and 1step off policy SARA are not exactly the same algorithm. But both are off policy and both follow the behavior policy. But in Q learning you need not consider the importance sampling ratio but in 1 step off policy SARSA you have to know both the behavior policy and target policy (since you have to find the sampling ratio). $\endgroup$ Apr 30 '20 at 7:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.