0
$\begingroup$

Here are SARSA and Q-learning from Sutton & Barto.

enter image description here

enter image description here

In these given forms, in my opinion, Q-learning is also on-policy because action selection is based on updated Q values. Where is my mistake, if any?

$\endgroup$
3
  • $\begingroup$ I found a very good answer here: stats.stackexchange.com/questions/184657/… $\endgroup$ Commented Apr 25 at 17:35
  • $\begingroup$ Does it fully answer your question or are there future questions? If it fully answers your question you might think of deleting the question as there's a high chance it gets flagged as duplicate right now. $\endgroup$
    – foreverska
    Commented Apr 25 at 17:54
  • $\begingroup$ Yes it answers but in my opinion it will be better if we keep this question here because its title is different: it is specific to Q-learning and SARSA. $\endgroup$ Commented Apr 25 at 18:14

1 Answer 1

2
$\begingroup$

Your mistake is your definition of on-policy. In both cases, we select actions based on our current action value estimates. It's how we select those actions that differentiates the 2 algorithms.


An algorithm is off-policy if the target policy is different from the exploration/behaviour policy.

An algorithm is on-policy if the 2 policies are the same.


I don't like these definitions because they hide details, which are required to understand the concepts. So, let me just explain why these algorithms are what they are.

In the case of Q-learning, which is off-policy, you try to learn a greedy policy, that is, a policy that always selects a greedy action, which is the action that leads to the highest amount of reward. That's why we use the max operator in Q-learning. The max operator selects the greedy action value with respect to the current value function estimate. Eventually, that value function estimate may be optimal. In Q-learning, we often also use the $\epsilon$-greedy policy for taking actions in the environment, but you could use any other exploration policy. So, there are 2 different policies. That's why Q-learning is off-policy.

In SARSA, in your case, we use $\epsilon$-greedy both to take actions in the environment and to update the formula.


Still, it's unclear where the target policy comes into play in these algorithms. Sure, we use the max to select the action reward in Q-learning, or we use the $\epsilon$-greedy policy to compute the target value, but why using the max operator, for instance, implies that we're learning the greedy policy? Once we have the optimal value function, we can just act greedily wrt it to get an optimal policy, which is greedy (Huh?). So, basically, we compute the target by assuming we would derive a specific policy from the value function and, in the tabular case, it turns out that this approach will find an optimal value function and policy.

$\endgroup$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .