The update rules for Q-learning and SARSA each are as follows:
Q Learning:
$$Q(s_t,a_t)←Q(s_t,a_t)+α[r_{t+1}+γ\max_{a'}Q(s_{t+1},a')−Q(s_t,a_t)]$$
SARSA:
$$Q(s_t,a_t)←Q(s_t,a_t)+α[r_{t+1}+γQ(s_{t+1},a_{t+1})−Q(s_t,a_t)]$$
I understand the theory that SARSA performs 'on-policy' updates, and Q-learning performs 'off-policy' updates.
At the moment I perform Q-learning by calculating the target thusly:
target = reward + self.y * np.max(self.action_model.predict(state_prime))
Here you can see I pick the maximum for the Q-function for state prime (i.e. greedy selection as defined by maxQ in the update rule). If I were to do a SARSA update and use the same on-policy as used when selecting an action, e.g. ϵ-greedy, would I basically change to this:
if np.random.random() < self.eps:
target = reward + self.y * self.action_model.predict(state_prime)[random.randint(0,9)]
else:
target = reward + self.y * np.max(self.action_model.predict(state_prime))
So sometimes it will pick a random future reward based on my epsilon greedy policy?