1
$\begingroup$

Here is the pseudocode for SARSA (which I took from here)

enter image description here

Are the two policies in SARSA for choosing an action equal? I guess yes, because it is called an on-policy learning algorithm. But could I, for example, also use different policies in this SARSA framework? For example an $\epsilon$-greedy policy and a softmax policy. Maybe the resulting algorithm would not be called SARSA anymore but it would be something similar.

$\endgroup$
3
  • $\begingroup$ I have deleted my answer as it caused too many follow-up questions, thus did not seem to be helping you. I highly recommend reading the book Reinforcement Learning: An Introduction, which covers the rationale behind basic value-based methods very well: incompleteideas.net/book/RLbook2020.pdf $\endgroup$ Dec 16 '21 at 18:08
  • $\begingroup$ @NeilSlater I think you should undelete your answer. It answers the question. If the OP is not able to understand it, it's a different matter. $\endgroup$
    – nbro
    Dec 16 '21 at 23:36
  • $\begingroup$ @nbro: OK, I have undeleted the question. $\endgroup$ Dec 17 '21 at 7:44
1
$\begingroup$

For learning, it doesn't matter much how you choose the first action before starting the main loop. That is because the agent doesn't need to learn about transitions to the first state of an episode.

The thing that does matter is that the first action choice should cover all possible actions with probability greater than zero, in order to guarantee convergence. Conceptually, this is not much different to using exploring starts.

However, the usual practice is to have just one active policy for SARSA, typically $\epsilon$-greedy based on Q values learned so far. It is worth noting here, that as Q values change during learning - which happens on each time step - then the policy may also change (when the greedy action choice changes). So even when you use the same rules to derive the policy in SARSA, the actual policy used may vary, even in the middle of the loop. In that respect, the SARSA algorithm uses many policies, but typically only one approach to determining the current policy.

If you are using function approximation and also used a very different rule for the policy for the first action, it is possible you could affect the function approximator through sample bias (your training data has different distribution to target data). This is tricky to put a number on in RL, but is usually ignored in off-policy approximation, so should not put you off if you want to try out ideas of using a different first time step policy.

$\endgroup$
4
  • $\begingroup$ Thanks Neil Slater. Unfortunately I have to admit that I have huge difficulties understanding it and I have several follow-up questions. 1) Why does "For learning, it doesn't matter much how you choose the first action before" ? Are you here refering to the first action of the whole algorithm or the first action selection in every step as we need to select 2 actions in SARSA in every step 2) Why does the policy in SARSA change when the Q-values are changing? 3) What do you mean by "So even when you use the same rules to derive the policy in SARSA"? What kind of rules are you talking about? $\endgroup$
    – PeterBe
    Dec 16 '21 at 17:38
  • $\begingroup$ 4) What do you mean by "[SARSA uses] typically only one approach to determining the current policy."? What do you mean by approach here? What is the difference between a policy, rules (Question 3) and approach here? 5) You write "the usual practice is to have just one active policy for SARSA". But is SARSA not an off-policy learning algorithm. So this should mean that it has to use at least 2 policies otherwise it would be on-policy learning as far as I see it. $\endgroup$
    – PeterBe
    Dec 16 '21 at 17:38
  • $\begingroup$ @PeterBe: I cannot answer your follow-up questions, sorry. You may find it helpful to review basic concepts in RL. I can recommend the book Sutton & Barto, Reinforcement Learning: An Introduction. You can get a copy at incompleteideas.net/book/RLbook2020.pdf and I think reading the first 4 or 5 chapters should answer your questions and be a lot more productive for you than more Q&A. $\endgroup$ Dec 17 '21 at 7:49
  • $\begingroup$ Do you know a comprehensive example (other than the book) where it is shown how the algorithm actually works. Any maybe you can try to answer the 2) and the 4) because they confused me most $\endgroup$
    – PeterBe
    Dec 17 '21 at 7:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.