SARSA is on-policy, while n-step SARSA is off-policy. But when n = 1, is it like an off-policy version of SARSA? Any similarity and difference between 1-step SARSA and SARSA?


1 Answer 1


N-step SARSA can be both off policy and on-policy. I think you already know the n step on-policy SARSA. So I am just telling you how n-step SARSA can be off-policy.

Off-policy n-step SARSA: Now you have two policies, one is target policy, $\pi$, (let's say it is a greedy policy), another one is behavior policy, $b$, (you are actually following this behavior policy). Since this is off-policy, you do importance sampling for that. So the update rule is like this:

$$Q_{t+n}(S_{t},A_{t}) = Q_{t+n-1}(S_{t},A_{t}) + \alpha \rho_{t+1:t+n-1}[G_{t:t+n} - Q_{t+n-1}(S_{t},A_{t})],$$


$$\rho_{t:h} = \prod_{t=k}^{h} \frac{\pi(A_{k}|S_{k})}{b(A_{k}|S_{k})}$$

You are following the behavior policy $b$, but shifting the Q values towards the target policy, $\pi$.

Off-policy one-step SARSA: You can think of Q learning as one-step off policy SARSA.

  • $\begingroup$ Thanks for your reply. Does that mean on-policy 1-step SARSA is the same as SARSA? $\endgroup$
    – ycenycute
    Apr 27, 2020 at 14:32
  • $\begingroup$ Yes, one-step on-policy SARSA is called SARSA $\endgroup$ Apr 27, 2020 at 15:13
  • $\begingroup$ Then what is the difference between Q-learning and 1-step off policy SARSA? I understand the algorithms are different. 1-step off policy SARSA uses importance sampling, and Q learning seems just following behavior policy. What would be different results applying these two algorithms? Are they equivalent theoretically? $\endgroup$
    – ycenycute
    Apr 29, 2020 at 21:11
  • $\begingroup$ Q learning and 1step off policy SARA are not exactly the same algorithm. But both are off policy and both follow the behavior policy. But in Q learning you need not consider the importance sampling ratio but in 1 step off policy SARSA you have to know both the behavior policy and target policy (since you have to find the sampling ratio). $\endgroup$ Apr 30, 2020 at 7:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .