Questions tagged [sarsa]

For questions related to the reinforcement learning (on-policy) algorithm called SARSA, which stands for (s, a, r, s', a').

Filter by
Sorted by
Tagged with
3
votes
1answer
105 views

How to determine if Q-learning has converged in practice?

I am using Q-learning and SARSA to solve a problem. The agent learns to go from the start to the goal without falling in the holes. At each state, I can choose the action corresponding to the maximum ...
3
votes
1answer
88 views

When do SARSA and Q-Learning converge to optimal Q values?

Here's another interesting multiple-choice question that puzzles me a bit. In tabular MDPs, if using a decision policy that visits all states an infinite number of times, and in each state, randomly ...
1
vote
0answers
24 views

Tic-tac-toe: How would standard SARSA and Q-learning yield different results in the agent's behaviour?

I know this is deceptively simple. Tic tac toe is a well studied game for RL. Assume your agent is playing aggainst a strong opponent. I know you deal in after states. I know that in Q learning the ...
0
votes
0answers
33 views

Why Q-Learning and SARSA have terrible performance?

I am trying to solve a MDP problem with almost 50 states and 60 actions with Q-Learning or SARSA. However, the performance of both algorithms is terrible and cannot find the optimal policy found by ...
2
votes
1answer
68 views

Implementing SARSA for a 2-stage Markov Decision Process

I am a bit confused as to how exactly I should be implementing SARSA (or Q-learning too) on what is a simple 2-stage Markov Decision Task. The structure of the task is as follows: Basically, there ...
0
votes
1answer
70 views

What are the differences between SARSA and Q-learning?

From Sutton and Barto's book Reinforcement Learning (Adaptive Computation and Machine Learning series), are the following definitions: To aid my learning of RL and gain an intuition, I'm focusing on ...
2
votes
1answer
44 views

Can the agent wait until the end of the episode to determine the reward in SARSA?

From Sutton and Barto's book Reinforcement Learning (Adaptive Computation and Machine Learning series) (p. 99), the following definition for first-visit MC prediction, for estimating $V \sim V_\pi$ is ...
8
votes
1answer
228 views

Are Q-learning and SARSA the same when action selection is greedy?

I'm currently studying reinforcement learning and I'm having difficulties with question 6.12 in Sutton and Barto's book. Suppose action selection is greedy. Is Q-learning then exactly the same ...
1
vote
1answer
77 views

What are the differences between 1-step SARSA and SARSA?

SARSA is on-policy, while n-step SARSA is off-policy. But when n = 1, is it like an off-policy version of SARSA? Any similarity and difference between 1-step SARSA and SARSA?
3
votes
1answer
226 views

Is Expected SARSA an off-policy or on-policy algorithm?

I understand that SARSA is an On-policy algorithm, and Q-learning an off-policy one. Sutton and Barto's textbook describes Expected Sarsa thusly: In these cliff walking results Expected Sarsa was ...
1
vote
1answer
65 views

What is the relationship between the Q and V functions?

Suppose we have a policy $\pi$ and we use SARSA to evaluate $Q^\pi(s, a)$, where $a$ is the policy $\pi$. Can we say that $Q^\pi(s, a) = V^\pi(s)$? The reason why I think this can be the case is ...
1
vote
2answers
48 views

What are the conditions for the convergence of SARSA to the optimal value function?

Is it correct that for SARSA to converge to the optimal value function (and policy) The learning rate parameter $\alpha$ must satisfy the conditions: $$\sum \alpha_{n^k(s,a)} =\infty \quad \text{and}...
1
vote
1answer
120 views

Why isn't my implementation of DQN using TensorFlow on the FrozenWorld environment working?

I am trying to test DQN on FrozenWorld environment in gym using TensorFlow 2.x. The update rule is (off policy) $$Q(s,a) \leftarrow Q(s,a)+\alpha (r+\gamma~ max_{a'}Q(s',a')-Q(s,a))$$ I am using an ...
2
votes
0answers
30 views

Evaluation a policy learned using Q - learning

I have been reading literature on reinforcement learning in healthcare. I am slightly confused between the policy evaluation for both SARSA and Q-learning. To my knowledge, I believe that SARSA is ...
2
votes
1answer
45 views

Do we need an explicit policy to sample $A'$ in order to compute the target in SARSA or Q-learning?

I would much appreciate if you could point me in the right direction regarding this question about targets for SARSA and Q-learning (notation: $S$ is the current state, $A$ is the current action, $R$ ...
1
vote
1answer
858 views

What is the difference between the epsilon greedy and softmax policies?

Could someone explain to me which is the key difference between the epsilon greedy policy and the softmax policy? In particular in the contest of SARSA and Q-Learning algorithms. I understood the main ...
1
vote
0answers
134 views

Expected SARSA, SARSA and Q-learning

I would much appreciate if you could point me in the right direction regarding this question about targets for approximate ...
3
votes
1answer
60 views

Optimal RL function approximation for TicTacToe game

I modeled the TicTacToe game as a RL problem - with an environment and an agent. At first I made an "Exact" agent - using the SARSA algorithm, I saved every unique state, and chose the best (...
3
votes
1answer
147 views

Understanding the n-step off-policy SARSA update

In Sutton & Barto's book (2nd ed) page 149, there is the equation 7.11 I am having a hard time understanding this equation. I would have thought that we should be moving $Q$ towards $G$, where $...
6
votes
1answer
2k views

Can Q-learning be used in a POMDP?

Can Q-learning (and SARSA) be directly used in a Partially Observable Markov Decision Process (POMDP)? If not, why not? My intuition is that the policies learned will be terrible because of partial ...
4
votes
1answer
2k views

Expected SARSA vs SARSA in “RL: An Introduction”

Sutton and Barto state in the 2018-version of "Reinforcement Learning: An Introduction" in the context of Expected SARSA (p. 133) the following sentences: Expected SARSA is more complex ...
3
votes
1answer
513 views

When does backward propagation occur in n-step SARSA?

I am trying to understand the algorithm for n-step SARSA from Sutton and Barto (2nd Edition). As I understand it, this algorithm should update n state-action values, but I cannot see where it is ...