0
$\begingroup$

Here is the pseudocode for SARSA (which I took from here)

enter image description here

Do we only select one action at the very beginning and then we always choose the same action for each step? Does it really make sense to choose the same initially chosen action $a$ regardless of the state $s$?

$\endgroup$
1
$\begingroup$

Do we only select one action at the very beginning and then we always choose the same action for each step?

No.

The pseudocode is clear on this, by using the word Choose and referencing a policy.

If you were expected to take the same action again, then the pseudocode already has the previous action in variable a, so it would not need to state anything about making a choice or using a policy.

The $a \leftarrow a'$ notation is common way to describe copying values*, so the variable a is changed at the end of each loop.

Does it really make sense to choose the same initially chosen action $a$ regardless of the state $s$?

Not in this case. Some learning algorithms do use a form of "sticky" exploration where a single exploratory action is committed to for multiple time steps. It can be useful in some environments. But not basic SARSA as described in the question.


* It avoids the ambiguity of $=$ as assignment or equality operator. You may also see $:=$ for assignment as an alternative, but for instance Sutton & Barto use $\leftarrow$ consistently, and a lot of RL literature follows this convention.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.