I know this is deceptively simple. Tic tac toe is a well studied game for RL.
Assume your agent is playing aggainst a strong opponent.
I know you deal in after states. I know that in Q learning the optimal policy should be converged on faster as the Q(S,A) is becoming closer to the optimal each step. While in SARSA the Q function will not be updated towards the optimal sometimes as it is exploring. If epislon is fixed SARSA will convereg to the epislon greedy policy.
I came across the question above and I don't know the answer, is it that SARSA may play more conservatively? Opting for more draws rather than getting in board positions where one could is likely to either lose or win rather than draw.