1
$\begingroup$

Here is the pseudocode for SARSA (which I took from here)

enter image description here

Are the two policies in SARSA for choosing an action equal? I guess yes, because it is called an on-policy learning algorithm. But could I, for example, also use different policies in this SARSA framework? For example an $\epsilon$-greedy policy and a softmax policy. Maybe the resulting algorithm would not be called SARSA anymore but it would be something similar.

Improve this question
$\endgroup$
3
  • $\begingroup$ I have deleted my answer as it caused too many follow-up questions, thus did not seem to be helping you. I highly recommend reading the book Reinforcement Learning: An Introduction, which covers the rationale behind basic value-based methods very well: incompleteideas.net/book/RLbook2020.pdf $\endgroup$
    – Neil Slater
    Dec 16, 2021 at 18:08
  • $\begingroup$ @NeilSlater I think you should undelete your answer. It answers the question. If the OP is not able to understand it, it's a different matter. $\endgroup$
    – nbro
    Dec 16, 2021 at 23:36
  • $\begingroup$ @nbro: OK, I have undeleted the question. $\endgroup$
    – Neil Slater
    Dec 17, 2021 at 7:44

1 Answer 1

Reset to default
1
$\begingroup$

For learning, it doesn't matter much how you choose the first action before starting the main loop. That is because the agent doesn't need to learn about transitions to the first state of an episode.

The thing that does matter is that the first action choice should cover all possible actions with probability greater than zero, in order to guarantee convergence. Conceptually, this is not much different to using exploring starts.

However, the usual practice is to have just one active policy for SARSA, typically $\epsilon$-greedy based on Q values learned so far. It is worth noting here, that as Q values change during learning - which happens on each time step - then the policy may also change (when the greedy action choice changes). So even when you use the same rules to derive the policy in SARSA, the actual policy used may vary, even in the middle of the loop. In that respect, the SARSA algorithm uses many policies, but typically only one approach to determining the current policy.

If you are using function approximation and also used a very different rule for the policy for the first action, it is possible you could affect the function approximator through sample bias (your training data has different distribution to target data). This is tricky to put a number on in RL, but is usually ignored in off-policy approximation, so should not put you off if you want to try out ideas of using a different first time step policy.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks Neil Slater. Unfortunately I have to admit that I have huge difficulties understanding it and I have several follow-up questions. 1) Why does "For learning, it doesn't matter much how you choose the first action before" ? Are you here refering to the first action of the whole algorithm or the first action selection in every step as we need to select 2 actions in SARSA in every step 2) Why does the policy in SARSA change when the Q-values are changing? 3) What do you mean by "So even when you use the same rules to derive the policy in SARSA"? What kind of rules are you talking about? $\endgroup$
    – PeterBe
    Dec 16, 2021 at 17:38
  • $\begingroup$ 4) What do you mean by "[SARSA uses] typically only one approach to determining the current policy."? What do you mean by approach here? What is the difference between a policy, rules (Question 3) and approach here? 5) You write "the usual practice is to have just one active policy for SARSA". But is SARSA not an off-policy learning algorithm. So this should mean that it has to use at least 2 policies otherwise it would be on-policy learning as far as I see it. $\endgroup$
    – PeterBe
    Dec 16, 2021 at 17:38
  • $\begingroup$ @PeterBe: I cannot answer your follow-up questions, sorry. You may find it helpful to review basic concepts in RL. I can recommend the book Sutton & Barto, Reinforcement Learning: An Introduction. You can get a copy at incompleteideas.net/book/RLbook2020.pdf and I think reading the first 4 or 5 chapters should answer your questions and be a lot more productive for you than more Q&A. $\endgroup$
    – Neil Slater
    Dec 17, 2021 at 7:49
  • $\begingroup$ Do you know a comprehensive example (other than the book) where it is shown how the algorithm actually works. Any maybe you can try to answer the 2) and the 4) because they confused me most $\endgroup$
    – PeterBe
    Dec 17, 2021 at 7:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .