# What are the differences between 1-step SARSA and SARSA?

SARSA is on-policy, while n-step SARSA is off-policy. But when n = 1, is it like an off-policy version of SARSA? Any similarity and difference between 1-step SARSA and SARSA?

N-step SARSA can be both off policy and on-policy. I think you already know the n step on-policy SARSA. So I am just telling you how n-step SARSA can be off-policy.

Off-policy n-step SARSA: Now you have two policies, one is target policy, $$\pi$$, (let's say it is a greedy policy), another one is behavior policy, $$b$$, (you are actually following this behavior policy). Since this is off-policy, you do importance sampling for that. So the update rule is like this:

$$Q_{t+n}(S_{t},A_{t}) = Q_{t+n-1}(S_{t},A_{t}) + \alpha \rho_{t+1:t+n-1}[G_{t:t+n} - Q_{t+n-1}(S_{t},A_{t})],$$

where

$$\rho_{t:h} = \prod_{t=k}^{h} \frac{\pi(A_{k}|S_{k})}{b(A_{k}|S_{k})}$$

You are following the behavior policy $$b$$, but shifting the Q values towards the target policy, $$\pi$$.

Off-policy one-step SARSA: You can think of Q learning as one-step off policy SARSA.

• Thanks for your reply. Does that mean on-policy 1-step SARSA is the same as SARSA? – Tracy Yang Apr 27 at 14:32
• Yes, one-step on-policy SARSA is called SARSA – Swakshar Deb Apr 27 at 15:13
• Then what is the difference between Q-learning and 1-step off policy SARSA? I understand the algorithms are different. 1-step off policy SARSA uses importance sampling, and Q learning seems just following behavior policy. What would be different results applying these two algorithms? Are they equivalent theoretically? – Tracy Yang Apr 29 at 21:11
• Q learning and 1step off policy SARA are not exactly the same algorithm. But both are off policy and both follow the behavior policy. But in Q learning you need not consider the importance sampling ratio but in 1 step off policy SARSA you have to know both the behavior policy and target policy (since you have to find the sampling ratio). – Swakshar Deb Apr 30 at 7:38