Questions tagged [sutton-barto]
For questions related to the book "Reinforcement Learning: An Introduction" (by Andrew Barto and Richard S. Sutton).
92
questions
0
votes
0
answers
23
views
Understanding SARSA with binary features and function approximation
I'm reading the Sutton and Barto Book on Reinforcement Learning (for oral exam preparation).
However, I'm stuck on the algorithm for SARSA($\lambda$) with binary features and linear function ...
- 1
1
vote
1
answer
85
views
Where is the problem: in batch TD(0) algorithm or in the code to solve AB problem in Sutton-Barto RL book?
Here is the batch TD(0) algorithm:
Here is the AB example I want to solve using batch TD(0):
And finally here is my Matlab code:
% eps1: A 0 B 0
% eps2: B 1
% eps3: B 1
% eps4: B 1
% eps5: B 1
% ...
- 285
1
vote
1
answer
74
views
How does off-policy Monte Carlo weighted importance sampling bias converge to zero (Sutton & Barto Section 5.5)
On Section 5.5 (page 105) of Sutton & Barto's "Reinforcement Learning: An Introduction", they discuss the off-policy Monte Carlo method for learning the value function of a target policy ...
- 198
5
votes
2
answers
107
views
Why is $\sum_{s} \eta(s)$ a constant of proportionality in the proof of the policy gradient theorem?
In Sutton and Barto's book (http://incompleteideas.net/book/bookdraft2017nov5.pdf), a proof of the policy gradient theorem is provided on pg. 269 for an episodic case and a start state policy ...
- 51
3
votes
1
answer
241
views
Sutton & Barto: what are parametrized functions?
From "Reinforcement Learning: An introduction (2nd ed.)" by Richard S. Sutton and Andrew G. Barto, on page 59
Instead, the agent would have to maintain $v_\pi$ and $q_\pi$ as parameterized ...
1
vote
1
answer
46
views
Why will every action be sampled an infinite number of times?
I am reading the book Reinforcement Learning: An Introduction. Second edition (Richard S. Sutton and Andrew G. Barto). In the k-armed bandit problem using $\varepsilon$-greedy selection method, the ...
- 13
1
vote
1
answer
136
views
In off-policy MC learning, why is the probability of sampling a trajectory the same as having a return?
In Sutton and Barto's RL book, in the section for off-policy learning, we would like to find the expected value of the random variable $G_t$, given $S_t = s$ under our target policy: $$\mathbb{E}_{\pi}...
- 13
3
votes
0
answers
59
views
How does off-policy monte carlo explore and converge? [duplicate]
Premises to question:
Behavior Policy: e-greedy (stochastic)
Target Policy: greedy (deterministic)
Importance Sampling Included
In off-policy Monte-Carlo control, the behavior policy chooses actions ...
- 31
2
votes
1
answer
31
views
What does it mean for an episode to start in a state-action pair?
In Sutton and Barto on chapter 5 (p.96), they talk about estimating state-action values with Monte Carlo:
For policy evaluation to work for action
values, we must assure continual exploration. One ...
- 91
3
votes
1
answer
97
views
Why does REINFORCE perform badly at first in Sutton and Barto Figure 13.1?
In Sutton and Barto (PDF, page 265), 2nd edition, Figure 13.1 applies REINFORCE to the "short corridor with switched actions" environment from Example 13.1. The figure looks like this:
My ...
- 133
1
vote
1
answer
48
views
What does "All store and access operations (for S(t) , A(t), and R(t)) can take their index mod n + 1" mean?
It's from the book Introduction to Reinforcement Learning. Second edition, chapter7: n-step Bootstrapping, page 147, n-step Sarsa.
I made the algo work, but I still don't understand the phrase.
...
- 93
0
votes
1
answer
71
views
Is my derivation of the Bellman equation for $q_{\pi}$ in terms of $p(s'|s,a)$ and $r(s,a)$ correct?
I have done exercise 3.29 from Sutton and Barto and I'd like to check if it's correct. Here's the exercise:
Rewrite the Bellman equation for the function $q_{\pi}$ in terms of the three argument ...
- 91
0
votes
0
answers
34
views
Is my derivation of the Bellman equation for $v_{\pi}$ in terms of $p(s'|s,a)$ and $r(s,a)$ correct?
I have exercise 3.29 from Sutton and Barto and I'd like to check if it's correct. Here's the exercise:
Rewrite the Bellman equation for the value function $v_{\pi}$ in terms of the three argument ...
- 91
0
votes
0
answers
28
views
Rewrite the four Bellman equations for the four value functions $(v_{\pi},v_*,q_{\pi},q_*)$ in terms of $p$ (3.4) and $r$ (3.5) [duplicate]
I have done exercise 3.29 from Sutton and Barto and I'd like to check if it's correct. Here's the exercise:
Rewrite the four Bellman equations for the four value functions $(v_{\pi},v_*,q_{\pi},q_*)$ ...
- 91
0
votes
1
answer
83
views
What is the equation for $\pi_*$ in terms of $q_*(s,a)$?
I am trying to solve the following exercise from Sutton and Barto:
Sutton and Barto Exercise 3.27 Give an equation for $\pi_*$ in terms of $q_*(s,a)$
However, I am struggling to do so. I know that $\...
- 91
0
votes
0
answers
32
views
Exercise 3.21 Sutton Barto: Draw or describe the contours of the optimal action-value function for putting, $q_{*}(s, putter)$, for the golf example
I am doing exercise 3.21 in Sutton and Barto. Here's the exercise:
Draw or describe the contours of the optimal action-value function for
putting, $q_{*}(s, putter)$, for the golf example.
Here's ...
- 91
4
votes
1
answer
82
views
$E_{\pi}[R_{t+1}|S_t=s,A_t=a] = E[R_{t+1}|S_t=s,A_t=a]$?
I would like to solve the first question of Exercise 3.19 from Sutton and Barto:
Exercise 3.19 The value of an action, $q_{\pi}(s, a)$, depends on the expected next reward and
the expected sum of the ...
- 91
7
votes
2
answers
1k
views
In Value Iteration, why can we initialize the value function arbitrarily?
I have not been able to find a good explanation of this, other than statements that the algorithm is guaranteed to converge with arbitrary choices for initial values in each state. Is this something ...
- 73
0
votes
1
answer
160
views
How to perform the back propagation step in Semi-gradient SARSA using a deep neural network?
For the back weight update step, I need to calculate $\nabla\hat{q}(S,A,w)$. My neural network takes in the state vector $S$ and gives out the action values for state $S$ and each action in the action ...
2
votes
1
answer
229
views
Why does the average-reward estimator for continuing tasks use the TD error?
In Sutton and Barto's RL book, section 10.3 describes how to use average reward $r(\pi)$ to define the quality of a policy, re-defining action-value function $q_\pi(s,a)$ and value function $v_\pi(s)$ ...
- 86
4
votes
1
answer
339
views
What should the discount factor for the non-slippery version of the FrozenLake environment be?
I was working with FrozenLake 4x4 from open AI gym. In the slippery case, using a discounting factor of 1, my value iteration implementation was giving a success rate of around 75 percent. It was much ...
1
vote
1
answer
27
views
If the probabilities with which each task is selected for you do not change over time, why would it appear as a single stationary k-armed bandit task?
Sutton-Barto (Section 2.9-Associative Search (Contextual Bandits), page 41):
As an example, suppose there are several different k-armed bandit tasks, and that on
each step you confront one of these ...
- 179
1
vote
1
answer
114
views
Gradient bandit algorithm: is $\bar{R}_t$ average of all rewards or average of rewards corresponding to $A_t$?
Sutton-Barto (Section 2.8-Gradient Bandit Algorithms, page 37):
Question: is $\bar{R}_t$ average of all rewards or average of rewards corresponding to $A_t$?
- 179
1
vote
0
answers
27
views
Knowing the futility of discounting in continuing problems, how can we say discounting has no role in control problems with function approximation?
Sutton-Barto (Section 10.4, page 254):
Based on the futility of discounting in continuing problems, how can we conclude that discounting has no role to play in control problems with function ...
- 179
1
vote
0
answers
31
views
Why is the step-size $\alpha_t = 1/t$ not appropriate for non-stationary function approximation?
Sutton-Barto (Section: Selecting Step-Size Parameters Manually, page: 222)
The classical choice $\alpha_t = 1/t$, which produces sample averages in tabular MC methods, is not appropriate for TD ...
- 179
0
votes
0
answers
24
views
What does "at one time" mean in the following question?
Sutton-Barto (Coarse Coding, page 216)
Example 9.3: Coarseness of Coarse Coding This example illustrates the e↵ect on
learning of the size of the receptive fields in coarse coding. Linear function ...
- 285
0
votes
0
answers
43
views
Proof of Convergence of Linear TD(0): Why do we need to show that each row sum plus the corresponding column sum is positive?
Sutton-Barto (page 206):
For our key matrix, $D(I − \gamma P)$, the diagonal entries are positive
and the off-diagonal entries are negative, so all we have to show is that each row
sum plus the ...
- 285
0
votes
0
answers
24
views
How can you know beforehand that the value of some actions are poor and what the greedy action is?
Sutton-Barto (Section 9.11, page 234)
The algorithms we have considered so far in this chapter have treated all the states
encountered equally, as if they were all equally important. In some cases, ...
- 285
0
votes
1
answer
42
views
Unclear paragraph in Sutton-Barto on "Tile Coding"
Sutton-Barto (Tile Coding, page 218):
For example, choosing $\alpha = 1/n$, where n is the number
of tilings, results in exact one-trial learning. If the example $s\to v$ is trained on, then
whatever ...
- 285
0
votes
1
answer
45
views
How state 1 has a 0.5 chance of terminating on the left, and state 950 has a 0.25 chance of terminating on the right?
Sutton-Barto's RL book (page 203)
Example 9.1: State Aggregation on the 1000-state Random Walk: Consider a
1000-state version of the random walk task (Examples 6.2 and 7.1 on pages 125 and
144). The ...
- 285
0
votes
1
answer
126
views
How is state aggregation defined mathematically here? [duplicate]
Sutton-Barto's RL book (page 203):
State aggregation is a simple form of generalizing function approximation in which
states are grouped together, with one estimated value (one component of the ...
- 285
0
votes
0
answers
22
views
What is meant by "weighting the update of each according to the on-policy distribution"?
In Sutton-Barto's RL book in Section 8.6 "Trajectory Sampling" (page 175), they say:
If one had an explicit representation
of the on-policy distribution, then one could sweep through all ...
- 285
1
vote
1
answer
43
views
Without planning, why does each episode only add one additional step to the policy?
In Sutton & Barto's RL book at page 165 for Example 8.1, they say:
Figure 8.3 shows why the planning agents found the solution so much faster than the nonplanning agent. Shown are the policies ...
- 285
1
vote
0
answers
57
views
Why does importance sampling ratio start and end one step later in off-policy SARSA given in Sutton-Barto's RL book?
In Sutton & Barto's RL book (page 149) they say:
Sarsa update can be completely replaced by a simple off-policy form
$Q_{t+n}(S_t,A_t)=Q_{t+n−1}(S_t,A_t) + \rho_{t+1:t+n} [G_{t:t+n} − Q_{t+n−1}(...
- 285
0
votes
1
answer
117
views
Why $ t=τ+n-1$ instead of $t=τ+n$ in n-step TD?
If $\tau$ is the time, whose state’s estimate is being updated, and $t$ is the current time, then, in n-step TD method, we have $t=\tau+n$ (because we have to wait n-steps, before we can update). ...
- 285
2
votes
1
answer
59
views
Why do we have $t$ as subscript in $V$ instead of $t+1$ in the expression of $G_{t:t+1}$?
In one-step TD updates, the target is the first reward plus the discounted estimated value of the next state, which we call the one-step return (page 143 of Sutton & Barto):
$$
G_{t:t+1} \...
- 285
1
vote
1
answer
122
views
Unclear step in off-policy (every-visit) MC Control: why do we need the line: $A_t \neq \pi(S_t)$ then exit inner loop?
Could please some expert on reinforcement learning explain the red-box part in the following off-policy MC control? I mean I did not understand what (and why)
is done in the step shown as a red-box. I ...
- 179
0
votes
2
answers
241
views
Why does OpenAI's PPO algorithm not follow the discounting method used in Sutton & Barto?
As discussed in this question, the policy gradient algorithms given in Reinforcement Learning: An Introduction use the gradient
\begin{align*}
\gamma^t \hat A_t \nabla_{\theta} \log \pi(a_t \, | \, ...
- 1,131
2
votes
0
answers
62
views
In Policy Gradient methods, why are actions always chosen from a Gaussian in the literature?
In Sutton's 2020 Reinforcement Learning text (in chapter 13.7 Policy Parameterization for Continuous Actions) it's stated
actions [may be] chosen from a normal (Gaussian) distribution.
However, I ...
- 121
4
votes
1
answer
278
views
What is the difference between an on-policy distribution and state visitation frequency?
On-policy distribution is defined as follows in Sutton and Barto:
On the other hand, state visitation frequency is defined as follows in Trust Region Policy Optimization:
$$\rho_{\pi}(s) = \sum_{t=0}^...
- 349
2
votes
1
answer
432
views
Why would SARSA diverge (but not Expected SARSA or Q-learning)?
In figure 6.3 (shown below) from Reinforcement Learning: An Introduction (second edition) by Sutton and Barto, SARSA is shown to perform worse asymptotically (after 100k episodes) than in the interim (...
- 121
0
votes
1
answer
71
views
Policies for which the policy improvement theorem holds
According to Reinforcement Learning (2nd Edition) by Sutton and Barto, the policy improvement theorem states that for any pair of deterministic policies $\pi'$ and $\pi$, if $q_\pi(s,\pi'(s)) \geq v_\...
2
votes
1
answer
45
views
How to prove importance sampling ratio is uncorrelated with action-value (or state-value) estimate?
In Sutton & Barto (2nd edition), the following is mentioned on page 150 (p. 172 of the pdf), section 7.4:
the importance sampling ratio has expected value one (Section 5.9) and is uncorrelated ...
- 349
0
votes
0
answers
23
views
Why does one-step TD strengthen only the last action of the sequence of actions that led to the high reward, while n-step TD the last n actions?
In the caption of figure 7.4 (p. 147) of Sutton & Barto's book (2nd edition), it's written
The one-step method strengthens only the last action of the sequence of actions that led to the high ...
- 349
2
votes
1
answer
180
views
In off-policy MC control algorithm by Sutton & Barto, why do we perform a last update when sample action is inconsistent with target policy?
I have a question about the $W$ term in the off-policy MC control algorithm on Page 111 of Sutton & Barto. I have also included it in the figure below.
My question: shouldn't the check $A_{t} = \...
- 123
6
votes
1
answer
218
views
If $\gamma \in (0,1)$, what is the on-policy state distribution for episodic tasks?
In Reinforcement Learning: An Introduction, section 9.2 (page 199), Sutton and Barto describe the on-policy distribution in episodic tasks, with $\gamma =1$, as being
\begin{equation}
\mu(s) = \frac{\...
- 103
2
votes
1
answer
104
views
Is the existence and uniqueness of the state-value function for $\gamma < 1$ theoretical?
Consider the following statement from 4.1 Policy Evaluation of the first edition of Sutton and Barto's book.
The existence and uniqueness of $V^{\pi}$ are guaranteed as long as
either $\gamma < 1$...
- 3,481
0
votes
1
answer
148
views
How to simplify policy gradient theorem to $E_{\pi}[G_t \frac{\nabla_{\theta}\pi(a|S_t,\theta)}{\pi(a|S_t,\theta)}]$?
In "Introduction to Reinforcement Learning" (Richard Sutton) section 13.3(Reinforce algorithm) they have the following equation:
\begin{align}
\nabla_{\theta}J &\propto \sum_s \mu(s) \...
- 663
1
vote
1
answer
75
views
Why is the update in-place faster than the out-of-place one in dynamic programming?
In Barto and Sutton's book, it's written that we have two types of updates in dynamic programming
Update out-of-place
Update in-place
The update in-place is the faster one. Why is that the case?
...
- 13
0
votes
0
answers
304
views
Suppose every-visit MC was used instead of first-visit MC on blackjack. Would you expect the results to be different?
This is a question from page 94 of Sutton and Barto's RL book 2020.
I read in someone's compiled GitHub answers to this book's exercises their answer was: "No because each state in an episode of ...
- 687