Suuton-Barto, page 176:

experiment to assess the effect empirically. To isolate the e↵ect of the update distribution, we used entirely one-step expected tabular updates, as defined by (8.1). In the uniform case, we cycled through all state–action pairs, updating each in place, and in the on-policy case we simulated episodes, all starting in the same state, updating each state–action pair that occurred under the current $\epsilon$-greedy policy ($\epsilon$=0.1). The tasks were undiscounted episodic tasks, generated randomly as follows. From each of the |S| states, two actions were possible, each of which resulted in one of b next states, all equally likely, with a different random selection of b states for each state–action pair. The branching factor, b, was the same for all state–action pairs. In addition, on all transitions there was a 0.1 probability of transition to the terminal state, ending the episode. The expected reward on each transition was selected from a Gaussian distribution with mean 0 and variance 1.

At any point in the planning process one can stop and exhaustively compute $v_{\tilde{\pi}}(s_0)$, the true value of the start state under the greedy policy, $\tilde{\pi}$, given the current action-value function Q, as an indication of how well the agent would do on a new episode on which it acted greedily (all the while assuming the model is correct).

Question: In the first paragraph, authors say the policy used for simulation is $\epsilon$-greedy and in the second paragraph, they are now specifying $\tilde{\pi}$ to be the greedy policy corresponding to the current action-value function Q. However, does not the current action-value function Q correspond to the $\epsilon$-greedy policy?

  • $\begingroup$ Note in the picture there are three red lines and three blue lines. Also, please don't paste screenshots full of text. Use text. you could then use pull quotes instead of referring to highlighted bits. The graph seems irrelevant to your question, but including it and referring to colours that are in it makes it appear relevant $\endgroup$ Commented May 17 at 15:57
  • $\begingroup$ I meant the underlined red and blue lines in the text, not red and blue lines in the plot. I updated this in the question. $\endgroup$ Commented May 18 at 17:09
  • $\begingroup$ Thanks. Still needs to be text, not a screenshot please. Screenshots of text from textbooks are not searchable on the site, and less easy for screenreaders to process. $\endgroup$ Commented May 18 at 22:20
  • $\begingroup$ I agree. Now, question statement is in text. $\endgroup$ Commented May 18 at 23:42

1 Answer 1


The text is about SARSA, so yes the action values in the Q table are estimates based on-policy, on the $\epsilon$-greedy policy used for learning, with a specific value of $\epsilon$.

However, this in no way constrains how the action values can or should be used to derive any other policy. Deriving a greedy policy from the current action values will typically be an improvement (as per the policy improvement theorem), and also a reasonable gauge of how well the agent has learned the environment so far.

It is fairly common approach in SARSA when using $\epsilon$-greedy policies to decay $\epsilon$, and for both Q-learning and SARSA to evaluate the current greedy policy at snapshots in order to track learning progress without the complication of allowing for exploratory actions.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .