1
$\begingroup$

I have some difficulties understanding the difference between Q-learning and SARSA. Here (What are the differences between SARSA and Q-learning?) the following updating formulas are given:

Q-Learning

$$Q(s,a) = Q(s,a) + \alpha (R_{t+1} + \gamma \max_aQ(s',a) - Q(s,a))$$

SARSA

$$Q(s,a) = Q(s,a) + \alpha (R_{t+1} + \gamma Q(s',a') - Q(s,a))$$

I know that SARSA is for on-policy learning while Q-learning is off-policy learning. So, in Q-learning, the epsilon-greedy policy (or epsilon-soft or softmax policy) is chosen for selecting the actions and the greedy policy is chosen for updating the Q-values. In SARSA the epsilon-greedy policy (or epsilon-soft or softmax policy) is chosen for selecting the actions and for updating the Q function.

So, actually, I have a question on that:

On this website (https://www.cse.unsw.edu.au/~cs9417ml/RL1/algorithms.html) there is written for SARSA

As you can see, there are two action selection steps needed, for determining the next state-action pair along with the first.

What is meant by two action selections? Normally you can only select one action per iteration. The other "selection" should be for the update.

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
3
$\begingroup$

In my view, the best way to understand these algorithms is to read the pseudocode (multiple times, if necessary!).

Here's the pseudocode of Q-learning.

enter image description here

Here's the pseudocode of SARSA.

enter image description here

So, as you can see, in SARSA, we choose one action before the episode starts, and, during the episode, we choose (and take) again more actions. In both cases, we choose these actions with the same policy (e.g. $\epsilon$-greedy), which is derived from $Q$. In Q-learning, we do not choose an action before the episode starts. We only choose and take an action at each step of the episode (like in SARSA). Hence, in SARSA, we choose actions in two places (but only take an action at each step of the episode). Note the difference between choosing/selecting an action and taking an action in the environment (you may just choose an action to update the Q-function, i.e. without taking it into the environment!).

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thanks for your answer nbro. Unfortunately I have to admit that I still have problems understanding it and I have several follow-up questions. 1) Why are we choosing more than 1 action in SARSA? One for going into the next state and the other one for updating the Q function? 2) In SARSA do we only select one action at the very beginning and then we always choose the same action for each step? Does it really make sense to choose the same initally choosen action a regardless of the sate s $\endgroup$
    – PeterBe
    Dec 14, 2021 at 15:00
  • $\begingroup$ 3) Are the two policies in SARSA for choosing an action equal? I guess yes, because it is called an on-policy learning algorithm. 4) How can we calculate the Q(s', a') in both SARSA and Q-learning for updating the Q-function. After having taken an action a at state s we get the reward r which we can observe. But we cannot observe Q(s',a') from the environment as far as I see it. $\endgroup$
    – PeterBe
    Dec 14, 2021 at 15:00
  • $\begingroup$ @PeterBe Create one post for each of these specific questions and I or someone else will answer them. Please, really only one question per post and provide the context for each question in their separate post. $\endgroup$
    – nbro
    Dec 14, 2021 at 16:28
  • $\begingroup$ Thanks for your answer and effort. I understand that generally each question should be asked in a separate post. However, at least the first 2 questions of mine are direct follow-up questions to your answer. It does not make sense to ask question 1) and 2) in a separate post, because I would exactly ask the same think as in this post (question 1) or react directly to something you mentioned in your answer (question 2). $\endgroup$
    – PeterBe
    Dec 14, 2021 at 16:32
  • $\begingroup$ @PeterBe In this answer, I am answering the question "What is meant by two action selections?" in the context of SARSA. This is actually different than explaining why we have 2 action selections in SARSA. If it's a follow-up question or not, it doesn't matter. Your questions could be useful for future readers, that's why I am asking you to ask each of your specific questions in their separate post. More people might have those same questions (but not exactly the same combination of questions: that's why it's important to ask each specific question in a separate post). $\endgroup$
    – nbro
    Dec 14, 2021 at 16:38

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .