Skip to main content
added 111 characters in body
Source Link
Neil Slater
  • 33.3k
  • 3
  • 44
  • 65

Why are we choosing more than 1 action in SARSA?

There is never a state where more than one action is chosen.

The appearance of two Choose statements is an artifact of the loop design and variable management in the pseudocode.

One for going into the next state and the other one for updating the Q function?

Sort of. There is only ever one chosen action for each state.

However, you need to have two actions in scope - the current action (just taken and observed) and planned next action, in order to process the update rule. Hence there are two variables a and a' and the code needs to generate one or other outside the main loop, or have some other way to ensure that it has access to current and next values. This is also why at the end of the loop, the "next" values $s', a'$ get copied to the "current" values $s, a$.

Why are we choosing more than 1 action in SARSA?

There is never a state where more than one action is chosen.

The appearance of two Choose statements is an artifact of the loop design and variable management in the pseudocode.

One for going into the next state and the other one for updating the Q function?

Sort of. There is only ever one chosen action for each state.

However, you need to have two actions in scope - the current action (just taken and observed) and planned next action, in order to process the update rule. Hence there are two variables a and a' and the code needs to generate one or other outside the main loop, or have some other way to ensure that it has access to current and next values.

Why are we choosing more than 1 action in SARSA?

There is never a state where more than one action is chosen.

The appearance of two Choose statements is an artifact of the loop design and variable management in the pseudocode.

One for going into the next state and the other one for updating the Q function?

Sort of. There is only ever one chosen action for each state.

However, you need to have two actions in scope - the current action (just taken and observed) and planned next action, in order to process the update rule. Hence there are two variables a and a' and the code needs to generate one or other outside the main loop, or have some other way to ensure that it has access to current and next values. This is also why at the end of the loop, the "next" values $s', a'$ get copied to the "current" values $s, a$.

Source Link
Neil Slater
  • 33.3k
  • 3
  • 44
  • 65

Why are we choosing more than 1 action in SARSA?

There is never a state where more than one action is chosen.

The appearance of two Choose statements is an artifact of the loop design and variable management in the pseudocode.

One for going into the next state and the other one for updating the Q function?

Sort of. There is only ever one chosen action for each state.

However, you need to have two actions in scope - the current action (just taken and observed) and planned next action, in order to process the update rule. Hence there are two variables a and a' and the code needs to generate one or other outside the main loop, or have some other way to ensure that it has access to current and next values.