All Questions
Tagged with barto-sutton or sutton-barto
23 questions with no upvoted or accepted answers
4
votes
0
answers
376
views
When using hashing in tile coding, why are memory requirements reduced and there is only a little loss of performance?
In the book "Reinforcement Learning: An Introduction" (2018) Sutton and Barto explain, on page 221, a form of tile coding using hashing, to reduce memory consumption.
I have two questions ...
- 321
4
votes
0
answers
64
views
Does everyone still use discount rates?
In Section 10.4 of Sutton and Barto's RL book, they argue that the discount rate $\gamma$ has no effect in continuing settings. They show (at least for one objective function) that the average of the ...
- 2,064
3
votes
0
answers
95
views
Sutton & Barto: Exercise 7.11 mistake?
Exercise from the book:
7.11 Show that if the approximate action values are unchanging, then the tree-backup return (7.16) can be written as a sum of expectation-based TD errors:
$$
\begin{align*}
&...
- 143
2
votes
1
answer
81
views
Is $s_0$ the current state in policy gradients?
As far as I understand from here (source: OpenAI), the objective function in Policy Gradient is:
$$J(\pi_{\theta})=E_{\tau\sim\pi_{\theta}}[R(\tau)],$$
where $R(\tau)=r_0+r_1+...+r_T$, with $r_t$ ...
2
votes
0
answers
122
views
In Policy Gradient methods, why are actions always chosen from a Gaussian in the literature?
In Sutton's 2020 Reinforcement Learning text (in chapter 13.7 Policy Parameterization for Continuous Actions) it's stated
actions [may be] chosen from a normal (Gaussian) distribution.
However, I ...
- 121
1
vote
1
answer
232
views
Where is the problem: in batch TD(0) algorithm or in the code to solve AB problem in Sutton-Barto RL book?
Here is the batch TD(0) algorithm:
Here is the AB example I want to solve using batch TD(0):
And finally here is my Matlab code:
% eps1: A 0 B 0
% eps2: B 1
% eps3: B 1
% eps4: B 1
% eps5: B 1
% ...
- 1,105
1
vote
1
answer
117
views
What does "All store and access operations (for S(t) , A(t), and R(t)) can take their index mod n + 1" mean?
It's from the book Introduction to Reinforcement Learning. Second edition, chapter7: n-step Bootstrapping, page 147, n-step Sarsa.
I made the algo work, but I still don't understand the phrase.
...
- 93
1
vote
0
answers
62
views
Knowing the futility of discounting in continuing problems, how can we say discounting has no role in control problems with function approximation?
Sutton-Barto (Section 10.4, page 254):
Based on the futility of discounting in continuing problems, how can we conclude that discounting has no role to play in control problems with function ...
- 199
1
vote
0
answers
43
views
Why is the step-size $\alpha_t = 1/t$ not appropriate for non-stationary function approximation?
Sutton-Barto (Section: Selecting Step-Size Parameters Manually, page: 222)
The classical choice $\alpha_t = 1/t$, which produces sample averages in tabular MC methods, is not appropriate for TD ...
- 199
1
vote
0
answers
105
views
Why does importance sampling ratio start and end one step later in off-policy SARSA given in Sutton-Barto's RL book?
In Sutton & Barto's RL book (page 149) they say:
Sarsa update can be completely replaced by a simple off-policy form
$Q_{t+n}(S_t,A_t)=Q_{t+n−1}(S_t,A_t) + \rho_{t+1:t+n} [G_{t:t+n} − Q_{t+n−1}(...
- 1,105
1
vote
2
answers
520
views
Why does OpenAI's PPO algorithm not follow the discounting method used in Sutton & Barto?
As discussed in this question, the policy gradient algorithms given in Reinforcement Learning: An Introduction use the gradient
\begin{align*}
\gamma^t \hat A_t \nabla_{\theta} \log \pi(a_t \, | \, ...
- 1,301
1
vote
1
answer
71
views
How to derive "value iteration" from "policy iteration"?
This is the equation for "value iteration" from Sutton-Barto:
\begin{align}
v_{k+1}(s)=& \max_{a \in \mathcal{A}}\mathbb{E} \Big(R_{t+1}+\gamma v_k(S_{t+1}) \big|S_t=s, A_t=a\Big) \\
=&...
- 1,105
0
votes
0
answers
113
views
What are the update equations for Double Expected Sarsa with an $\epsilon$-greedy target policy?
This is question 6.13 in Sutton-Barto,page 136.
What are the update equations for Double Expected Sarsa with an $\epsilon$-greedy target policy?
The answer is given as follows:
Let $Q_1$ and $Q_2$ ...
- 1,105
0
votes
0
answers
33
views
Confusing statement in Sutton-Barto on expected versus sample updates
Sutton-Barto, page 174.
b successor states are equally likely and in which the error in the initial estimate is 1. The values at the next states are assumed correct, so the expected update reduces
...
- 1,105
0
votes
0
answers
15
views
Is this a typo in n-step tree backup section in Sutton-Barto?
Sutton-Barto, page 153.
Should not it be $t<T-n$ in Eq.16? The reason is we have $t<T-1$ and $t<T-2$ for the 1 and 2 step returns, respectively.
- 1,105
0
votes
0
answers
16
views
Are these typos in n-step tree backup section in Sutton-Barto?
Sutton-Barto, page 153.
It seems to me that the "red" underlined parts are typos.
1-) 2-step tree backup return formula is valid for $t<T-2$ but the n-step version which includes $n\ge 2$...
- 1,105
0
votes
0
answers
16
views
Unclear point in n-step state value estimation
Sutton-Barto, page 143:
Here they say: "To make up for that, an equal number of additional updates
are made at the end of the episode, after termination and before starting the next episode.&...
- 1,105
0
votes
0
answers
40
views
Sutton-Barto confusing notation for the target and behaviour policy in the expected sarsa
Sutton-Barto, page 134, second paragraph:
In these cliff walking results Expected Sarsa was used on-policy, but in general it
might use a policy different from the target policy $\pi$ to generate ...
- 1,105
0
votes
0
answers
42
views
Derivation of update rule for Off-policy TD(0) with importance sampling ratio
Sutton-Barto, second edt, page 128, Exercise 6.7:
Design an off-policy version of the TD(0) update that can be used with
arbitrary target policy $\pi$ and covering behavior policy $b$, using at each ...
- 1,105
0
votes
0
answers
30
views
Pseudocode for batch TD(0)
This is from Sutton-Barto, second edt, page 126:
Suppose there is available only a finite amount of experience, say 10 episodes or 100
time steps. In this case, a common approach with incremental ...
- 1,105
0
votes
0
answers
27
views
What does the term "expected leaf node" in this exercise from Sutton-Barto mean?
What does the term "expected leaf node" in the Exercise below from Sutton-Barto mean?
- 1,105
0
votes
0
answers
29
views
Why does one-step TD strengthen only the last action of the sequence of actions that led to the high reward, while n-step TD the last n actions?
In the caption of figure 7.4 (p. 147) of Sutton & Barto's book (2nd edition), it's written
The one-step method strengthens only the last action of the sequence of actions that led to the high ...
- 379
0
votes
0
answers
552
views
Suppose every-visit MC was used instead of first-visit MC on blackjack. Would you expect the results to be different?
This is a question from page 94 of Sutton and Barto's RL book 2020.
I read in someone's compiled GitHub answers to this book's exercises their answer was: "No because each state in an episode of ...
- 805