How do updates in SARSA and Q-learning differ in code?

Question

The update rules for Q-learning and SARSA each are as follows:

Q Learning:

$$Q(s_t,a_t)←Q(s_t,a_t)+α[r_{t+1}+γ\max_{a'}Q(s_{t+1},a')−Q(s_t,a_t)]$$

SARSA:

$$Q(s_t,a_t)←Q(s_t,a_t)+α[r_{t+1}+γQ(s_{t+1},a_{t+1})−Q(s_t,a_t)]$$

I understand the theory that SARSA performs 'on-policy' updates, and Q-learning performs 'off-policy' updates.

At the moment I perform Q-learning by calculating the target thusly:

target = reward + self.y * np.max(self.action_model.predict(state_prime))

Here you can see I pick the maximum for the Q-function for state prime (i.e. greedy selection as defined by maxQ in the update rule). If I were to do a SARSA update and use the same on-policy as used when selecting an action, e.g. ϵ-greedy, would I basically change to this:

if np.random.random() < self.eps:
    target = reward + self.y * self.action_model.predict(state_prime)[random.randint(0,9)]
else:
    target = reward + self.y * np.max(self.action_model.predict(state_prime))

So sometimes it will pick a random future reward based on my epsilon greedy policy?

Answer 1

Picking actions and making updates should be treated as separate things. For Q-learning you also need to explore by using some exploration strategy (e.g. $\epsilon$-greedy).

Steps for Q-learning:

1. initialize state $S$
   For every step of the episode:
2. choose action $A$ by some exploratory policy (e.g. $\epsilon$-greedy) from state $S$
3. take action $A$ and observe $R$ and $S'$
4. do the update $Q(S, A) = Q(S, A) + \alpha(R + \gamma*\max_aQ(S', a) - Q(S, A))$
5. update the state $S = S'$ and keep looping from step 2 until the end of episode

Steps for Sarsa:

1. initialize state $S$
2. initialize first action $A$ from state $S$ by some exploratory policy (e.g. $\epsilon$-greedy)
   For every step of the episode:
3. take action $A$ and observe $R$ and $S'$
4. choose action $A'$ from state $S'$ by some exploratory policy (e.g. $\epsilon$-greedy)
5. do the update $Q(S, A) = Q(S, A) + \alpha(R + \gamma * Q(S', A') - Q(S, A))$
6. update state and action $S = S'$, $A = A'$ and keep looping from step 3 until end of the episode