All Questions
Tagged with barto-sutton or sutton-barto
121 questions
1
vote
1
answer
117
views
What does "All store and access operations (for S(t) , A(t), and R(t)) can take their index mod n + 1" mean?
It's from the book Introduction to Reinforcement Learning. Second edition, chapter7: n-step Bootstrapping, page 147, n-step Sarsa.
I made the algo work, but I still don't understand the phrase.
...
CommunityBot
- 1
1
vote
1
answer
232
views
Where is the problem: in batch TD(0) algorithm or in the code to solve AB problem in Sutton-Barto RL book?
Here is the batch TD(0) algorithm:
Here is the AB example I want to solve using batch TD(0):
And finally here is my Matlab code:
% eps1: A 0 B 0
% eps2: B 1
% eps3: B 1
% eps4: B 1
% eps5: B 1
% ...
CommunityBot
- 1
1
vote
2
answers
520
views
Why does OpenAI's PPO algorithm not follow the discounting method used in Sutton & Barto?
As discussed in this question, the policy gradient algorithms given in Reinforcement Learning: An Introduction use the gradient
\begin{align*}
\gamma^t \hat A_t \nabla_{\theta} \log \pi(a_t \, | \, ...
CommunityBot
- 1
7
votes
2
answers
635
views
Understanding the n-step off-policy SARSA update
In Sutton & Barto's book (2nd ed) page 149, there is the equation 7.11
I am having a hard time understanding this equation.
I would have thought that we should be moving $Q$ towards $G$, where $...
- 119
2
votes
1
answer
81
views
Is $s_0$ the current state in policy gradients?
As far as I understand from here (source: OpenAI), the objective function in Policy Gradient is:
$$J(\pi_{\theta})=E_{\tau\sim\pi_{\theta}}[R(\tau)],$$
where $R(\tau)=r_0+r_1+...+r_T$, with $r_t$ ...
CommunityBot
- 1
3
votes
3
answers
389
views
$\gamma^t$ in REINFORCE update (Sutton-Barto RL book Exercise 13.2)
I've struggled with solving exercise 13.2 from Reinforcement Learning: An Introduction Second Edition :
Generalize the box on page 199, the policy gradient theorem (13.5),
the proof of the policy ...
- 11
1
vote
2
answers
107
views
How are these two terms equivalent in Sutton and Barto's derivation of the REINFORCE algorithm
After reading Sutton and Barto, I was able to understand the derivation of this theorem. The only thing I don't get is the following part from REINFORCE algorithm:
How are these terms equivalent, and ...
- 41.4k
3
votes
2
answers
323
views
In off-policy MC control algorithm by Sutton & Barto, why do we perform a last update when sample action is inconsistent with target policy?
I have a question about the $W$ term in the off-policy MC control algorithm on Page 111 of Sutton & Barto. I have also included it in the figure below.
My question: shouldn't the check $A_{t} = \...
1
vote
1
answer
72
views
Confusing convention in Sutto-Barto on Monte Carlo Tree Search: is a leaf node a state leaf node or state-action leaf node?
Figure 8.10: Monte Carlo Tree Search. When the environment changes to a new state, MCTS
executes as many iterations as possible before an action needs to be selected, incrementally building a tree ...
- 1,703
-2
votes
1
answer
37
views
Unclear line in prioritized sweeping algorithm [closed]
Could someone explain the red line (especially, the meaning of the difference) in prioritized sweeping algorithm below?
Sutton-Barto, page 170:
- 1,557
0
votes
1
answer
57
views
Confusing statement in Sutton-Barto on trajectory sampling
Suuton-Barto, page 176:
experiment to assess the effect empirically. To isolate the e↵ect of the update distribution,
we used entirely one-step expected tabular updates, as defined by (8.1). In the ...
- 33.3k
0
votes
0
answers
113
views
What are the update equations for Double Expected Sarsa with an $\epsilon$-greedy target policy?
This is question 6.13 in Sutton-Barto,page 136.
What are the update equations for Double Expected Sarsa with an $\epsilon$-greedy target policy?
The answer is given as follows:
Let $Q_1$ and $Q_2$ ...
- 1,105
0
votes
0
answers
33
views
Confusing statement in Sutton-Barto on expected versus sample updates
Sutton-Barto, page 174.
b successor states are equally likely and in which the error in the initial estimate is 1. The values at the next states are assumed correct, so the expected update reduces
...
- 1,105
0
votes
2
answers
77
views
Why is dynamic programming an example of planning?
Sutton-Barto, page 160, towards bottom:
Why is dynamic programming an example of planning? There is no simulation in dynamic programming.
- 33.3k
0
votes
0
answers
15
views
Is this a typo in n-step tree backup section in Sutton-Barto?
Sutton-Barto, page 153.
Should not it be $t<T-n$ in Eq.16? The reason is we have $t<T-1$ and $t<T-2$ for the 1 and 2 step returns, respectively.
- 1,105
0
votes
1
answer
21
views
Unclear arrow in general Dyna architecture
Sutton-Barto, page 163:
Figure 8.1: The general Dyna Architecture. Real experience, passing back and forth between
the environment and the policy, affects policy and value functions in much the same ...
- 33.3k
0
votes
2
answers
43
views
Confusing points in Dyna-Q in Sutton-Barto about model, simulated experience and model predictions
Sutton-Barto, page 164:
In the pseudocode algorithm for
Dyna-Q in the box below, Model(s, a) denotes the contents of the model (predicted
next state and reward) for state–action pair (s, a). Direct ...
- 33.3k
0
votes
1
answer
17
views
Confusing point point in Dyna-Q
Sutton-Barto, page 164:
In the (f) loop, $S,A, S', R$ are from real experience (Model(S,A)=(R, S') where R and S' are also real expereince).
This experience is used in direct RL part in (d). Why is ...
- 1,557
0
votes
1
answer
22
views
Unclear points in Dyna Maze example in Sutton-Barto
Sutton-Barto, page 164:
The main part of Figure 8.2 shows average learning curves from an experiment in
which Dyna-Q agents were applied to the maze task. The initial action values were zero,
the ...
- 1,557
0
votes
1
answer
54
views
What is the backed-up value in dynamic programming and the corresponding update based on this backed up value?
Sutton-Barto, page 160:
Dynamic programming methods clearly fit this structure: they make sweeps through the
space of states, generating for each state the distribution of possible transitions. Each
...
- 33.3k
0
votes
0
answers
16
views
Are these typos in n-step tree backup section in Sutton-Barto?
Sutton-Barto, page 153.
It seems to me that the "red" underlined parts are typos.
1-) 2-step tree backup return formula is valid for $t<T-2$ but the n-step version which includes $n\ge 2$...
- 1,105
0
votes
1
answer
55
views
Why no falling off cliff in SARSA for the example in Sutton-Barto?
Sutton-Barto, page 132:
The graph to the right shows the
performance of the Sarsa and Qlearning
methods with "-greedy action
selection, " = 0.1. After an
initial transient, Q-learning ...
- 41.4k
0
votes
0
answers
16
views
Unclear point in n-step state value estimation
Sutton-Barto, page 143:
Here they say: "To make up for that, an equal number of additional updates
are made at the end of the episode, after termination and before starting the next episode.&...
- 1,105
0
votes
0
answers
40
views
Sutton-Barto confusing notation for the target and behaviour policy in the expected sarsa
Sutton-Barto, page 134, second paragraph:
In these cliff walking results Expected Sarsa was used on-policy, but in general it
might use a policy different from the target policy $\pi$ to generate ...
- 1,105
3
votes
0
answers
95
views
Sutton & Barto: Exercise 7.11 mistake?
Exercise from the book:
7.11 Show that if the approximate action values are unchanging, then the tree-backup return (7.16) can be written as a sum of expectation-based TD errors:
$$
\begin{align*}
&...
- 143
-1
votes
1
answer
57
views
Suppose action selection is greedy. Is Q-learning then exactly the same algorithm as Sarsa?
Below is the Exercise 6.12 from Sutto-Barto and its solution (from the solution manual) but I was not able to understand it. I will be happy if one can make it clearer.
- 2,632
1
vote
1
answer
41
views
Unclear sentence in Sutton-Barto in Temporal-Difference chapter
Sutton-Barto, Chapter 6, page:130
By 8000 time steps, the greedy policy was long since optimal (a trajectory from it is shown
inset); continued $\epsilon$-greedy exploration kept the average episode ...
- 1,557
0
votes
0
answers
42
views
Derivation of update rule for Off-policy TD(0) with importance sampling ratio
Sutton-Barto, second edt, page 128, Exercise 6.7:
Design an off-policy version of the TD(0) update that can be used with
arbitrary target policy $\pi$ and covering behavior policy $b$, using at each ...
- 1,105
0
votes
0
answers
30
views
Pseudocode for batch TD(0)
This is from Sutton-Barto, second edt, page 126:
Suppose there is available only a finite amount of experience, say 10 episodes or 100
time steps. In this case, a common approach with incremental ...
- 1,105
1
vote
1
answer
54
views
How does the Belman optimality equation with altered transition probabilities in the second equation follow?
Sutton-Barto, page 102 (second edition).
How does the Belman optimality with altered transition probabilities in the second equation follow? The point which confuses me is the first part inside the ...
- 5,422
2
votes
1
answer
260
views
Where does the TD formula for tic-tac-toe in Sutton & Barto come from?
In section $1.5$ of the book "Reinforcement Learning: An Introduction" by Sutton and Barto they use tic-tac-toe as an example of an RL use case. They provide the following temporal ...
- 41.4k
1
vote
2
answers
45
views
Why is the better policy defined with respect to all the states values being greater?
In Sutton & Barto (Section 3.6 - Optimal Policies and Optimal Value Functions), they say that :
Value functions define a partial ordering over policies. A policy $\pi$ is defined to be better ...
- 41.4k
2
votes
1
answer
41
views
Why is there the potential problem of "learning only from the tails of episodes" in off-policy MC control?
Sutton-Barto page 111, first paragraph (Off-policy Monte Carlo Control):
A potential problem is that this method learns only from the tails of episodes, when
all of the remaining actions in the ...
- 41.4k
0
votes
1
answer
53
views
In Sutton-Barto a confusing point regarding $\epsilon$-soft policies in the proof for optimality of MC control without exploring starts
Sutton-Barto, page 102:
In the second paragraph, we have:
Consider a new environment that is just like the original environment,
except with the requirement that policies be $\epsilon$-soft “moved
...
- 166
3
votes
2
answers
104
views
Confusing point in Sutton-Barto: replacing $a$ in $q(s,a)$ with a stochastic policy $\pi^\prime$
Sutton-Barto, page 101, Eq (5.2):
Assume that $\pi^\prime$ is the $\epsilon$-greedy policy. Then,
\begin{align}
q_{\pi}\big(s,\pi'(s)\big)&= \sum_{a}\pi'(a|s)q_{\pi}(s,a) \\
...
- 33.3k
1
vote
1
answer
71
views
How to derive "value iteration" from "policy iteration"?
This is the equation for "value iteration" from Sutton-Barto:
\begin{align}
v_{k+1}(s)=& \max_{a \in \mathcal{A}}\mathbb{E} \Big(R_{t+1}+\gamma v_k(S_{t+1}) \big|S_t=s, A_t=a\Big) \\
=&...
- 5,422
1
vote
1
answer
82
views
Unclear point in derivation of action-value function [closed]
I did not understand how third equality follows from the second equality. Could some expert explain?
- 5,422
4
votes
2
answers
261
views
$E_{\pi}[R_{t+1}|S_t=s,A_t=a] = E[R_{t+1}|S_t=s,A_t=a]$?
I would like to solve the first question of Exercise 3.19 from Sutton and Barto:
Exercise 3.19 The value of an action, $q_{\pi}(s, a)$, depends on the expected next reward and
the expected sum of the ...
- 1,105
0
votes
0
answers
27
views
What does the term "expected leaf node" in this exercise from Sutton-Barto mean?
What does the term "expected leaf node" in the Exercise below from Sutton-Barto mean?
- 1,105
1
vote
1
answer
70
views
If the probabilities with which each task is selected for you do not change over time, why would it appear as a single stationary k-armed bandit task?
Sutton-Barto (Section 2.9-Associative Search (Contextual Bandits), page 41):
As an example, suppose there are several different k-armed bandit tasks, and that on
each step you confront one of these ...
- 1,040
1
vote
1
answer
110
views
Unclear paragraph in Sutton-Barto on exploration/exploitation relating to bandit like decision tasks [closed]
This is a text from Sutton-Barto, second edition, page 30:
Even if the underlying task is stationary and deterministic,the learner faces a set of bandit like decision tasks each of which changes over ...
- 1,040
0
votes
1
answer
104
views
How can I get Q-Learning (1 step off policy) update from n-step off policy learning update?
In Sutton and Barto we have expressions for Q-Learning and n-step Off policy learning. The former ought to be the 1-step limit of the latter but I cannot see it working out that way. What am I missing?...
- 210
1
vote
1
answer
79
views
Sutton & Barto: Why expected square of the importance-sampling-scaled return is for policy b?
From "Reinforcement Learning: An introduction (2nd ed.)" by Richard S. Sutton and Andrew G. Barto, on page 107 and 108:
We can verify that the variance of the importance-sampling scaled
...
- 33.3k
5
votes
2
answers
202
views
Why is $\sum_{s} \eta(s)$ a constant of proportionality in the proof of the policy gradient theorem?
In Sutton and Barto's book (http://incompleteideas.net/book/bookdraft2017nov5.pdf), a proof of the policy gradient theorem is provided on pg. 269 for an episodic case and a start state policy ...
CommunityBot
- 1
4
votes
1
answer
126
views
Could you explain these 2 steps of the derivation of the Bellman equation as a recursive equation in Sutton & Barto?
I am reading the Sutton & Barto (2018) RL textbook.
On page 59, it derives the recursive property of the state-value function as below.
Could you explain the steps of third and fourth equality?
...
- 227
2
votes
1
answer
362
views
Unclear step in off-policy (every-visit) MC Control: why do we need the line: $A_t \neq \pi(S_t)$ then exit inner loop?
Could please some expert on reinforcement learning explain the red-box part in the following off-policy MC control? I mean I did not understand what (and why)
is done in the step shown as a red-box. I ...
- 332
2
votes
1
answer
312
views
How does off-policy Monte Carlo weighted importance sampling bias converge to zero (Sutton & Barto Section 5.5)
On Section 5.5 (page 105) of Sutton & Barto's "Reinforcement Learning: An Introduction", they discuss the off-policy Monte Carlo method for learning the value function of a target policy ...
- 248
5
votes
1
answer
856
views
What is the difference between an on-policy distribution and state visitation frequency?
On-policy distribution is defined as follows in Sutton and Barto:
On the other hand, state visitation frequency is defined as follows in Trust Region Policy Optimization:
$$\rho_{\pi}(s) = \sum_{t=0}^...
- 41.4k
3
votes
2
answers
461
views
In the on-policy state distribution for episodic tasks, why don't we take into account the length of the episode?
In Sutton & Barto's "Reinforcement Learning: An Introduction", 2nd edition, page 199, they describe the on-policy distribution for episodic tasks in the following box:
I don't understand how this ...
- 41.4k
3
votes
0
answers
60
views
How does off-policy monte carlo explore and converge? [duplicate]
Premises to question:
Behavior Policy: e-greedy (stochastic)
Target Policy: greedy (deterministic)
Importance Sampling Included
In off-policy Monte-Carlo control, the behavior policy chooses actions ...
- 41.4k