If we write the pseudo-code for the SARSA algorithm we first initialise our hyper-parameters etc. and then initialise $S_t$, which we use to choose $A_t$ from our policy $\pi(a|s)$. Then for each $t$ in the episode we do the following:
- Take action $A_t$ and observe $R_{t+1}$, $S_{t+1}$
- Choose $A_{t+1}$ using $S_{t+1}$ in our policy
- $Q(S_t, A_t) = Q(S_t, A_t) + \alpha [R_{t+1} + \gamma Q(S_{t+1},A_{t+1}) - Q(S_t, A_t)]$
Now, in Q-learning we replace $Q(S_{t+1},A_{t+1})$ in line 3 with $\max_aQ(S_{t+1},a)$. Recall that in SARSA we chose our $A_{t+1}$ using our policy $\pi$ - if our policy is greedy with respect to the action value function then this simply means the policy is $\pi(a|s) = \text{argmax}_aQ(s,a)$ which is exactly how we choose our weight update in Q-learning.
To answer the question - no, they are not always the same algorithm.
Consider where we transition from $s$ to $s'$ where $s'=s$. I will outline the updates for SARSA and Q-learning indexing the $Q$ functions with $t$ to demonstrate the difference.
For each case, I will assume we are at the start of the episode, as this is the easiest way to illustrate the difference. Note that actions denoted by $A_i$ are for actions taken explicitly in the environment -- in the Q-Learning update the $\max$ action that is chosen for the update is not executed in the environment, the action taken in the environment is chosen by the policy after the update has happened.
SARSA
- We initialise $S_0 = s$ and choose $A_0 = \text{argmax}_a Q_0(s,a)$
- Take action $A_0$ and observe $R_{1}$ and $S_{1} = s' = s$.
- Choose action $A_{1} = \text{argmax}_aQ_{0}(s,a)$
- $Q_{1}(S_0,A_0) = Q_0(S_0,A_0) + \alpha [R_{1} + \gamma Q_0(s,A_1) - Q_0(S_0,A_0)]$
Q-Learning
- Initialise $S_0 = s$
- Choose action $A_0 = \text{argmax}_aQ_0(s,a)$, observe $R_{1}, S_{1} = s' = s$
- $Q_{1}(S_0,A_0) = Q_0(S_0,A_0) + \alpha [R_{1} + \gamma \max_aQ_0(s,a) - Q_0(S_0,A_0)]$
- Choose action $A_1 = \text{argmax}_aQ_1(s,a)$
As you can see the next action for the updates in SARSA (line 4) and Q-learning (line 3) are taken with respect to the same $Q$ function, but the key difference is that the actual next action taken in $Q$-learning is taken with respect to the updated $Q$-function.
The key for understanding this edge case is that when we transition into the same state, the Q-Learning update will update the Q-function before choosing $A_1$. I have indexed actions and Q-functions by the episode step - hopefully, it makes sense why I have done this for the Q-functions as, usually, this would not make sense, but, because we have two successive states that are the same, it is okay.