Questions tagged [sarsa]
For questions related to the reinforcement learning (on-policy) algorithm called SARSA, which stands for (s, a, r, s', a').
45
questions
0
votes
1
answer
65
views
It is not clear why SARSA is on-policy but Q-learning off-policy
Here are SARSA and Q-learning from Sutton & Barto.
In these given forms, in my opinion, Q-learning is also on-policy because action selection is based on updated Q values. Where is my mistake, if ...
- 974
0
votes
0
answers
49
views
What else could I try to avoid catastrophic forgetting in my implementation of Semi-Gradient SARSA?
I was trying to implement the semi-gradient SARSA algorithm (see p.244 Reinforcement Learning: An Introduction) using PyTorch:
...
- 195
3
votes
1
answer
251
views
Convergence of epsilon greedy policy (with no epsilon decay) using TD Learning?
If I create a policy using the q-values of an epsilon greedy policy using the Sarsa algorithm (not changing the epsilon with each episode), will it converge to the optimal solution to the MDP? I am ...
0
votes
1
answer
191
views
Help on Deep Sarsa algorithm that work with pytorch (Adam optimiser) but not with keras/Tensorflow (Adam optimiser)
I have a deep sarsa algorithm wich work great on Pytorch on lunar-lander-v2 and I would use with Keras/Tensorflow. It use mini-batch of size 64 wich are used 128 time to train at each episode.
There ...
- 1
0
votes
1
answer
405
views
How to perform the back propagation step in Semi-gradient SARSA using a deep neural network?
For the back weight update step, I need to calculate $\nabla\hat{q}(S,A,w)$. My neural network takes in the state vector $S$ and gives out the action values for state $S$ and each action in the action ...
2
votes
1
answer
847
views
Is Q-learning only capable of learning a deterministic policy?
I was following a reinforcement learning course on coursera and in this video at 2:57 the instructor says
Expected SARSA and SARSA both allow us to learn an optimal
$\epsilon$-soft policy, but, Q-...
3
votes
1
answer
108
views
Can we also estimate $V_{\pi}$ with SARSA?
For SARSA, I know we can estimate the action value $Q(s,a)$, and the relationship between $V(s)$ and $Q(s,a)$ is $V_{\pi}(s) = \sum_{a \in \mathcal{A}} \pi(a|s)Q_{\pi}
(s,a)$.
So my question is, can ...
- 31
2
votes
1
answer
410
views
Is it possible learning convergence is lost in Reinforcement Learning as the state space grows?
I am new in the AI field and I am trying to use Reinforcement Learning. Specifically, I am using tabular Q-Learning and SARSA algorithms to solve a sequential decision making problem. (I am using <...
- 33
1
vote
0
answers
98
views
Why does importance sampling ratio start and end one step later in off-policy SARSA given in Sutton-Barto's RL book?
In Sutton & Barto's RL book (page 149) they say:
Sarsa update can be completely replaced by a simple off-policy form
$Q_{t+n}(S_t,A_t)=Q_{t+n−1}(S_t,A_t) + \rho_{t+1:t+n} [G_{t:t+n} − Q_{t+n−1}(...
- 974
4
votes
2
answers
671
views
Is the optimal policy the one with the highest accumulative reward (Q-Learning vs SARSA)?
I was looking at the following diagram,
The reward obtained with SARSA is higher. However, the path that Q learning chooses is eventually the optimal one, isn't it? Why is the SARSA reward higher if ...
- 282
1
vote
1
answer
624
views
How is $Q(s', a')$ calculated in SARSA and Q-Learning?
I have a question about how to update the Q-function in Q-learning and SARSA. Here (What are the differences between SARSA and Q-learning?) the following updating formulas are given:
Q-Learning
$$Q(s,...
- 256
1
vote
1
answer
150
views
Are the two policies in SARSA for choosing an action the same?
Here is the pseudocode for SARSA (which I took from here)
Are the two policies in SARSA for choosing an action equal? I guess yes, because it is called an on-policy learning algorithm. But could I, ...
- 256
0
votes
1
answer
49
views
Are we choosing the same action in every step in SARSA?
Here is the pseudocode for SARSA (which I took from here)
Do we only select one action at the very beginning and then we always choose the same action for each step? Does it really make sense to ...
- 256
0
votes
1
answer
58
views
Why are we choosing more than 1 action in SARSA?
Here is the pseudocode for SARSA (which I took from here)
Why are we choosing more than 1 action in SARSA? One for going into the next state and the other one for updating the Q function?
- 256
1
vote
1
answer
364
views
What is meant by "two action selections" in SARSA?
I have some difficulties understanding the difference between Q-learning and SARSA. Here (What are the differences between SARSA and Q-learning?) the following updating formulas are given:
Q-Learning
$...
- 256
2
votes
0
answers
197
views
Reduction of state space of the game Connect Four to apply RL algorithms SARSA and Q-Learning
I would like to implement the reinforcement learning algorithms SARSA and Q-Learning for the board game Connect Four.
I am familiar with the algorithms and know about their limitations regarding large ...
- 21
2
votes
1
answer
773
views
Why would SARSA diverge (but not Expected SARSA or Q-learning)?
In figure 6.3 (shown below) from Reinforcement Learning: An Introduction (second edition) by Sutton and Barto, SARSA is shown to perform worse asymptotically (after 100k episodes) than in the interim (...
- 121
0
votes
0
answers
81
views
Reinforcement learning for rearranging the mobile home screen icon layout: what inputs/states do I need to pass into the algorithm?
I have a problem where I need to rearrange a particular user's mobile home screen icon layout. Let's say that the social media app usage of a user is high compared to other app usage. So I need the ...
- 51
4
votes
1
answer
219
views
How should I generate datasets for a SARSA agent when the environment is not simple?
I am currently working on my master's thesis and going to apply Deep-SARSA as my DRL algorithm. The problem is that there is no datasets available and I guess that I should generate them somehow. ...
- 153
2
votes
0
answers
85
views
Can $Q$-learning or SARSA be thought of a Markov Chain?
I might just be overthinking a very simple question but nonetheless the following has been bugging me a lot.
Given an MDP with non-trivial state and action sets, we can implement the SARSA algorithm ...
- 182
2
votes
0
answers
118
views
Is Deep SARSA learning a feasible approach?
I noticed that SARSA has been rarely used in the deep RL setting. Usually, the training for DQN is done off-policy. I think one of the major reasons for this is due to greater sample efficiency in ...
- 1,291
2
votes
1
answer
72
views
Intuitively, how does it make sense to take an action $A'$ when the environment already ended? [duplicate]
The update equation for SARSA is $Q(S,A) = R + \gamma Q(S',A')$. Consider this: I take an action $A$ that leads to the terminal state. Now my $S'$ would be one of the terminal states. So...
...
- 41
3
votes
1
answer
2k
views
How to determine if Q-learning has converged in practice？
I am using Q-learning and SARSA to solve a problem. The agent learns to go from the start to the goal without falling in the holes.
At each state, I can choose the action corresponding to the maximum ...
- 31
3
votes
1
answer
361
views
How are we calculating the average reward ($r(\pi)$) if the policy changes over time?
In the average reward setting, the quality of a policy is defined as:
$$ r(\pi) = \lim_{h\to\infty}\frac{1}{h} \sum_{j=1}^{h}E[R_j] $$
When we reach the steady state distribution, we can write the ...
- 693
4
votes
1
answer
1k
views
When do SARSA and Q-Learning converge to optimal Q values?
Here's another interesting multiple-choice question that puzzles me a bit.
In tabular MDPs, if using a decision policy that visits all states an infinite number of times, and in each state, randomly ...
- 1,161
2
votes
0
answers
187
views
Tic-tac-toe: How would standard SARSA and Q-learning yield different results in the agent's behaviour?
I know this is deceptively simple. Tic tac toe is a well studied game for RL.
Assume your agent is playing aggainst a strong opponent.
I know you deal in after states. I know that in Q learning the ...
- 121
2
votes
1
answer
290
views
Implementing SARSA for a 2-stage Markov Decision Process
I am a bit confused as to how exactly I should be implementing SARSA (or Q-learning too) on what is a simple 2-stage Markov Decision Task. The structure of the task is as follows:
Basically, there ...
- 182
2
votes
1
answer
812
views
What are the differences between SARSA and Q-learning? [closed]
From Sutton and Barto's book Reinforcement Learning (Adaptive Computation and Machine Learning series), are the following definitions:
To aid my learning of RL and gain an intuition, I'm focusing on ...
- 335
3
votes
1
answer
2k
views
Can the agent wait until the end of the episode to determine the reward in SARSA?
From Sutton and Barto's book Reinforcement Learning (Adaptive Computation and Machine Learning series) (p. 99), the following definition for first-visit MC prediction, for estimating $V \sim V_\pi$ is ...
- 335
11
votes
2
answers
2k
views
Are Q-learning and SARSA the same when action selection is greedy?
I'm currently studying reinforcement learning and I'm having difficulties with question 6.12 in Sutton and Barto's book.
Suppose action selection is greedy. Is Q-learning then exactly the same ...
- 131
1
vote
1
answer
950
views
What are the differences between 1-step SARSA and SARSA?
SARSA is on-policy, while n-step SARSA is off-policy. But when n = 1, is it like an off-policy version of SARSA? Any similarity and difference between 1-step SARSA and SARSA?
- 351
6
votes
1
answer
3k
views
Is Expected SARSA an off-policy or on-policy algorithm?
I understand that SARSA is an On-policy algorithm, and Q-learning an off-policy one.
Sutton and Barto's textbook describes Expected Sarsa thusly:
In these cliff walking results Expected Sarsa was ...
- 63
1
vote
1
answer
399
views
What is the relationship between the Q and V functions?
Suppose we have a policy $\pi$ and we use SARSA to evaluate $Q^\pi(s, a)$, where $a$ is the policy $\pi$.
Can we say that $Q^\pi(s, a) = V^\pi(s)$?
The reason why I think this can be the case is ...
- 1,291
1
vote
2
answers
2k
views
What are the conditions for the convergence of SARSA to the optimal value function?
Is it correct that for SARSA to converge to the optimal value function (and policy)
The learning rate parameter $\alpha$ must satisfy the conditions:
$$\sum \alpha_{n^k(s,a)} =\infty \quad \text{and}...
- 309
2
votes
1
answer
471
views
Why isn't my implementation of DQN using TensorFlow on the FrozenWorld environment working?
I am trying to test DQN on FrozenWorld environment in gym using TensorFlow 2.x. The update rule is (off policy)
$$Q(s,a) \leftarrow Q(s,a)+\alpha (r+\gamma~ max_{a'}Q(s',a')-Q(s,a))$$
I am using an ...
- 121
3
votes
0
answers
101
views
Evaluation a policy learned using Q - learning
I have been reading literature on reinforcement learning in healthcare. I am slightly confused between the policy evaluation for both SARSA and Q-learning.
To my knowledge, I believe that SARSA is ...
- 1,291
2
votes
1
answer
91
views
Do we need an explicit policy to sample $A'$ in order to compute the target in SARSA or Q-learning?
I would much appreciate if you could point me in the right direction regarding this question about targets for SARSA and Q-learning (notation: $S$ is the current state, $A$ is the current action, $R$ ...
- 123
3
votes
1
answer
8k
views
What is the difference between the $\epsilon$-greedy and softmax policies?
Could someone explain to me which is the key difference between the $\epsilon$-greedy policy and the softmax policy? In particular, in the contest of SARSA and Q-Learning algorithms. I understood the ...
- 199
3
votes
1
answer
217
views
Optimal RL function approximation for TicTacToe game
I modeled the TicTacToe game as a RL problem - with an environment and an agent.
At first I made an "Exact" agent - using the SARSA algorithm, I saved every unique state, and chose the best (...
- 422
6
votes
2
answers
594
views
Understanding the n-step off-policy SARSA update
In Sutton & Barto's book (2nd ed) page 149, there is the equation 7.11
I am having a hard time understanding this equation.
I would have thought that we should be moving $Q$ towards $G$, where $...
- 163
10
votes
1
answer
6k
views
Can Q-learning be used in a POMDP?
Can Q-learning (and SARSA) be directly used in a Partially Observable Markov Decision Process (POMDP)? If not, why not? My intuition is that the policies learned will be terrible because of partial ...
- 298
5
votes
1
answer
4k
views
Expected SARSA vs SARSA in "RL: An Introduction" [closed]
Sutton and Barto state in the 2018-version of "Reinforcement Learning: An Introduction" in the context of Expected SARSA (p. 133) the following sentences:
Expected SARSA is more complex ...
- 311
3
votes
1
answer
2k
views
How to deal with the terminal state in SARSA in a multi-agent setting?
I'm training a SARSA agent to update a Q function, but I'm confused about how you handle the final state. In this case, when the game ends and there is no $S'$.
For example, the agent performed an ...
- 403
8
votes
2
answers
3k
views
How should I handle action selection in the terminal state when implementing SARSA?
I recently started learning about reinforcement learning. Currently, I am trying to implement the SARSA algorithm. However, I do not know how to deal with $Q(s', a')$, when $s'$ is the terminal state. ...
- 562
3
votes
1
answer
595
views
When does backward propagation occur in n-step SARSA?
I am trying to understand the algorithm for n-step SARSA from Sutton and Barto (2nd Edition). As I understand it, this algorithm should update n state-action values, but I cannot see where it is ...
- 131