Confusing statement in Sutton-Barto on trajectory sampling

Question

Suuton-Barto, page 176:

experiment to assess the effect empirically. To isolate the e↵ect of the update distribution, we used entirely one-step expected tabular updates, as defined by (8.1). In the uniform case, we cycled through all state–action pairs, updating each in place, and in the on-policy case we simulated episodes, all starting in the same state, updating each state–action pair that occurred under the current $\epsilon$-greedy policy ($\epsilon$=0.1). The tasks were undiscounted episodic tasks, generated randomly as follows. From each of the |S| states, two actions were possible, each of which resulted in one of b next states, all equally likely, with a different random selection of b states for each state–action pair. The branching factor, b, was the same for all state–action pairs. In addition, on all transitions there was a 0.1 probability of transition to the terminal state, ending the episode. The expected reward on each transition was selected from a Gaussian distribution with mean 0 and variance 1.

At any point in the planning process one can stop and exhaustively compute $v_{\tilde{\pi}}(s_0)$, the true value of the start state under the greedy policy, $\tilde{\pi}$, given the current action-value function Q, as an indication of how well the agent would do on a new episode on which it acted greedily (all the while assuming the model is correct).

Question: In the first paragraph, authors say the policy used for simulation is $\epsilon$-greedy and in the second paragraph, they are now specifying $\tilde{\pi}$ to be the greedy policy corresponding to the current action-value function Q. However, does not the current action-value function Q correspond to the $\epsilon$-greedy policy?

Answer 1

The text is about SARSA, so yes the action values in the Q table are estimates based on-policy, on the $\epsilon$-greedy policy used for learning, with a specific value of $\epsilon$.

However, this in no way constrains how the action values can or should be used to derive any other policy. Deriving a greedy policy from the current action values will typically be an improvement (as per the policy improvement theorem), and also a reasonable gauge of how well the agent has learned the environment so far.

It is fairly common approach in SARSA when using $\epsilon$-greedy policies to decay $\epsilon$, and for both Q-learning and SARSA to evaluate the current greedy policy at snapshots in order to track learning progress without the complication of allowing for exploratory actions.