Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [sutton-barto]

For questions related to the book "Reinforcement Learning: An Introduction" (by Andrew Barto and Richard S. Sutton).

Filter by
Sorted by
Tagged with
0 votes
0 answers
10 views

Confusing convention in Sutto-Barto on Monte Carlo Tree Search: is a leaf node a state leaf node or state-action leaf node?

Figure 8.10: Monte Carlo Tree Search. When the environment changes to a new state, MCTS executes as many iterations as possible before an action needs to be selected, incrementally building a tree ...
DSPinfinity's user avatar
-1 votes
1 answer
19 views

Unclear line in prioritized sweeping algorithm

Could someone explain the red line (especially, the meaning of the difference) in prioritized sweeping algorithm below? Sutton-Barto, page 170:
DSPinfinity's user avatar
0 votes
0 answers
8 views

For simulated experience, Rollout algorithms = classical MC control?

Rollout algorithms are decision-time planning algorithms based on Monte Carlo control applied to simulated trajectories (using a model) that all begin at the current state. Does that mean for ...
DSPinfinity's user avatar
0 votes
0 answers
33 views

What are the update equations for Double Expected Sarsa with an $\epsilon$-greedy target policy?

This is question 6.13 in Sutton-Barto,page 136. What are the update equations for Double Expected Sarsa with an $\epsilon$-greedy target policy? The answer is given as follows: Let $Q_1$ and $Q_2$ ...
DSPinfinity's user avatar
0 votes
1 answer
39 views

Confusing statement in Sutton-Barto on trajectory sampling

Suuton-Barto, page 176: experiment to assess the effect empirically. To isolate the e↵ect of the update distribution, we used entirely one-step expected tabular updates, as defined by (8.1). In the ...
DSPinfinity's user avatar
0 votes
0 answers
27 views

Confusing statement in Sutton-Barto on expected versus sample updates

Sutton-Barto, page 174. b successor states are equally likely and in which the error in the initial estimate is 1. The values at the next states are assumed correct, so the expected update reduces ...
DSPinfinity's user avatar
0 votes
2 answers
65 views

Why is dynamic programming an example of planning?

Sutton-Barto, page 160, towards bottom: Why is dynamic programming an example of planning? There is no simulation in dynamic programming.
DSPinfinity's user avatar
0 votes
0 answers
12 views

Is this a typo in n-step tree backup section in Sutton-Barto?

Sutton-Barto, page 153. Should not it be $t<T-n$ in Eq.16? The reason is we have $t<T-1$ and $t<T-2$ for the 1 and 2 step returns, respectively.
DSPinfinity's user avatar
0 votes
2 answers
22 views

Confusing points in Dyna-Q in Sutton-Barto about model, simulated experience and model predictions

Sutton-Barto, page 164: In the pseudocode algorithm for Dyna-Q in the box below, Model(s, a) denotes the contents of the model (predicted next state and reward) for state–action pair (s, a). Direct ...
DSPinfinity's user avatar
0 votes
1 answer
19 views

Unclear points in Dyna Maze example in Sutton-Barto

Sutton-Barto, page 164: The main part of Figure 8.2 shows average learning curves from an experiment in which Dyna-Q agents were applied to the maze task. The initial action values were zero, the ...
DSPinfinity's user avatar
0 votes
1 answer
17 views

Confusing point point in Dyna-Q

Sutton-Barto, page 164: In the (f) loop, $S,A, S', R$ are from real experience (Model(S,A)=(R, S') where R and S' are also real expereince). This experience is used in direct RL part in (d). Why is ...
DSPinfinity's user avatar
0 votes
1 answer
19 views

Unclear arrow in general Dyna architecture

Sutton-Barto, page 163: Figure 8.1: The general Dyna Architecture. Real experience, passing back and forth between the environment and the policy, affects policy and value functions in much the same ...
DSPinfinity's user avatar
0 votes
1 answer
25 views

What is the backed-up value in dynamic programming and the corresponding update based on this backed up value?

Sutton-Barto, page 160: Dynamic programming methods clearly fit this structure: they make sweeps through the space of states, generating for each state the distribution of possible transitions. Each ...
DSPinfinity's user avatar
0 votes
0 answers
14 views

Are these typos in n-step tree backup section in Sutton-Barto?

Sutton-Barto, page 153. It seems to me that the "red" underlined parts are typos. 1-) 2-step tree backup return formula is valid for $t<T-2$ but the n-step version which includes $n\ge 2$...
DSPinfinity's user avatar
0 votes
1 answer
39 views

Why no falling off cliff in SARSA for the example in Sutton-Barto?

Sutton-Barto, page 132: The graph to the right shows the performance of the Sarsa and Qlearning methods with "-greedy action selection, " = 0.1. After an initial transient, Q-learning ...
DSPinfinity's user avatar
0 votes
0 answers
15 views

Unclear point in n-step state value estimation

Sutton-Barto, page 143: Here they say: "To make up for that, an equal number of additional updates are made at the end of the episode, after termination and before starting the next episode.&...
DSPinfinity's user avatar
-1 votes
0 answers
20 views

In what cases options framework results in faster learning in RL?

If we define options in such a way that any optimal trajectory from starting point to the goal, you never visit a state that belongs to the set of states that are in any of these options. In what ...
ijnooihoikhjuihnn's user avatar
0 votes
0 answers
32 views

Sutton-Barto confusing notation for the target and behaviour policy in the expected sarsa

Sutton-Barto, page 134, second paragraph: In these cliff walking results Expected Sarsa was used on-policy, but in general it might use a policy different from the target policy $\pi$ to generate ...
DSPinfinity's user avatar
3 votes
0 answers
80 views

Sutton & Barto: Exercise 7.11 mistake?

Exercise from the book: 7.11 Show that if the approximate action values are unchanging, then the tree-backup return (7.16) can be written as a sum of expectation-based TD errors: $$ \begin{align*} &...
Max Gorbunov's user avatar
-1 votes
1 answer
36 views

Suppose action selection is greedy. Is Q-learning then exactly the same algorithm as Sarsa?

Below is the Exercise 6.12 from Sutto-Barto and its solution (from the solution manual) but I was not able to understand it. I will be happy if one can make it clearer.
DSPinfinity's user avatar
1 vote
1 answer
39 views

Unclear sentence in Sutton-Barto in Temporal-Difference chapter

Sutton-Barto, Chapter 6, page:130 By 8000 time steps, the greedy policy was long since optimal (a trajectory from it is shown inset); continued $\epsilon$-greedy exploration kept the average episode ...
DSPinfinity's user avatar
0 votes
0 answers
24 views

Derivation of update rule for Off-policy TD(0) with importance sampling ratio

Sutton-Barto, second edt, page 128, Exercise 6.7: Design an off-policy version of the TD(0) update that can be used with arbitrary target policy $\pi$ and covering behavior policy $b$, using at each ...
DSPinfinity's user avatar
0 votes
0 answers
19 views

Pseudocode for batch TD(0)

This is from Sutton-Barto, second edt, page 126: Suppose there is available only a finite amount of experience, say 10 episodes or 100 time steps. In this case, a common approach with incremental ...
DSPinfinity's user avatar
1 vote
1 answer
39 views

How does the Belman optimality equation with altered transition probabilities in the second equation follow?

Sutton-Barto, page 102 (second edition). How does the Belman optimality with altered transition probabilities in the second equation follow? The point which confuses me is the first part inside the ...
DSPinfinity's user avatar
2 votes
1 answer
231 views

Where does the TD formula for tic-tac-toe in Sutton & Barto come from?

In section $1.5$ of the book "Reinforcement Learning: An Introduction" by Sutton and Barto they use tic-tac-toe as an example of an RL use case. They provide the following temporal ...
mNugget's user avatar
  • 167
1 vote
2 answers
42 views

Why is the better policy defined with respect to all the states values being greater?

In Sutton & Barto (Section 3.6 - Optimal Policies and Optimal Value Functions), they say that : Value functions define a partial ordering over policies. A policy $\pi$ is defined to be better ...
pew31's user avatar
  • 13
2 votes
1 answer
33 views

Why is there the potential problem of "learning only from the tails of episodes" in off-policy MC control?

Sutton-Barto page 111, first paragraph (Off-policy Monte Carlo Control): A potential problem is that this method learns only from the tails of episodes, when all of the remaining actions in the ...
DSPinfinity's user avatar
0 votes
1 answer
42 views

In Sutton-Barto a confusing point regarding $\epsilon$-soft policies in the proof for optimality of MC control without exploring starts

Sutton-Barto, page 102: In the second paragraph, we have: Consider a new environment that is just like the original environment, except with the requirement that policies be $\epsilon$-soft “moved ...
DSPinfinity's user avatar
3 votes
2 answers
71 views

Confusing point in Sutton-Barto: replacing $a$ in $q(s,a)$ with a stochastic policy $\pi^\prime$

Sutton-Barto, page 101, Eq (5.2): Assume that $\pi^\prime$ is the $\epsilon$-greedy policy. Then, \begin{align} q_{\pi}\big(s,\pi'(s)\big)&= \sum_{a}\pi'(a|s)q_{\pi}(s,a) \\ ...
DSPinfinity's user avatar
1 vote
1 answer
53 views

How to derive "value iteration" from "policy iteration"?

This is the equation for "value iteration" from Sutton-Barto: \begin{align} v_{k+1}(s)=& \max_{a \in \mathcal{A}}\mathbb{E} \Big(R_{t+1}+\gamma v_k(S_{t+1}) \big|S_t=s, A_t=a\Big) \\ =&...
DSPinfinity's user avatar
1 vote
1 answer
55 views

Unclear point in derivation of action-value function [closed]

I did not understand how third equality follows from the second equality. Could some expert explain?
DSPinfinity's user avatar
0 votes
0 answers
25 views

What does the term "expected leaf node" in this exercise from Sutton-Barto mean?

What does the term "expected leaf node" in the Exercise below from Sutton-Barto mean?
DSPinfinity's user avatar
1 vote
1 answer
105 views

Unclear paragraph in Sutton-Barto on exploration/exploitation relating to bandit like decision tasks [closed]

This is a text from Sutton-Barto, second edition, page 30: Even if the underlying task is stationary and deterministic,the learner faces a set of bandit like decision tasks each of which changes over ...
DSPinfinity's user avatar
2 votes
1 answer
66 views

Is $s_0$ the current state in policy gradients?

As far as I understand from here (source: OpenAI), the objective function in Policy Gradient is: $$J(\pi_{\theta})=E_{\tau\sim\pi_{\theta}}[R(\tau)],$$ where $R(\tau)=r_0+r_1+...+r_T$, with $r_t$ ...
fermented_bean's user avatar
1 vote
1 answer
65 views

Sutton & Barto: Why expected square of the importance-sampling-scaled return is for policy b?

From "Reinforcement Learning: An introduction (2nd ed.)" by Richard S. Sutton and Andrew G. Barto, on page 107 and 108: We can verify that the variance of the importance-sampling scaled ...
Max Gorbunov's user avatar
3 votes
2 answers
258 views

$\gamma^t$ in REINFORCE update (Sutton-Barto RL book Exercise 13.2)

I've struggled with solving exercise 13.2 from Reinforcement Learning: An Introduction Second Edition : Generalize the box on page 199, the policy gradient theorem (13.5), the proof of the policy ...
cfml's user avatar
  • 53
4 votes
1 answer
96 views

Could you explain these 2 steps of the derivation of the Bellman equation as a recursive equation in Sutton & Barto?

I am reading the Sutton & Barto (2018) RL textbook. On page 59, it derives the recursive property of the state-value function as below. Could you explain the steps of third and fourth equality? ...
tesio's user avatar
  • 205
1 vote
1 answer
173 views

Where is the problem: in batch TD(0) algorithm or in the code to solve AB problem in Sutton-Barto RL book?

Here is the batch TD(0) algorithm: Here is the AB example I want to solve using batch TD(0): And finally here is my Matlab code: % eps1: A 0 B 0 % eps2: B 1 % eps3: B 1 % eps4: B 1 % eps5: B 1 % ...
DSPinfinity's user avatar
2 votes
1 answer
244 views

How does off-policy Monte Carlo weighted importance sampling bias converge to zero (Sutton & Barto Section 5.5)

On Section 5.5 (page 105) of Sutton & Barto's "Reinforcement Learning: An Introduction", they discuss the off-policy Monte Carlo method for learning the value function of a target policy ...
user118967's user avatar
5 votes
2 answers
168 views

Why is $\sum_{s} \eta(s)$ a constant of proportionality in the proof of the policy gradient theorem?

In Sutton and Barto's book (http://incompleteideas.net/book/bookdraft2017nov5.pdf), a proof of the policy gradient theorem is provided on pg. 269 for an episodic case and a start state policy ...
jwl17's user avatar
  • 59
0 votes
1 answer
92 views

How can I get Q-Learning (1 step off policy) update from n-step off policy learning update?

In Sutton and Barto we have expressions for Q-Learning and n-step Off policy learning. The former ought to be the 1-step limit of the latter but I cannot see it working out that way. What am I missing?...
Borun Chowdhury's user avatar
3 votes
1 answer
292 views

Sutton & Barto: what are parametrized functions?

From "Reinforcement Learning: An introduction (2nd ed.)" by Richard S. Sutton and Andrew G. Barto, on page 59 Instead, the agent would have to maintain $v_\pi$ and $q_\pi$ as parameterized ...
SomeoneUnknown's user avatar
1 vote
1 answer
55 views

Why will every action be sampled an infinite number of times?

I am reading the book Reinforcement Learning: An Introduction. Second edition (Richard S. Sutton and Andrew G. Barto). In the k-armed bandit problem using $\varepsilon$-greedy selection method, the ...
k2pctdn's user avatar
  • 55
1 vote
1 answer
228 views

In off-policy MC learning, why is the probability of sampling a trajectory the same as having a return?

In Sutton and Barto's RL book, in the section for off-policy learning, we would like to find the expected value of the random variable $G_t$, given $S_t = s$ under our target policy: $$\mathbb{E}_{\pi}...
ArminAshrafi's user avatar
3 votes
0 answers
60 views

How does off-policy monte carlo explore and converge? [duplicate]

Premises to question: Behavior Policy: e-greedy (stochastic) Target Policy: greedy (deterministic) Importance Sampling Included In off-policy Monte-Carlo control, the behavior policy chooses actions ...
Jonah Kim's user avatar
2 votes
1 answer
39 views

What does it mean for an episode to start in a state-action pair?

In Sutton and Barto on chapter 5 (p.96), they talk about estimating state-action values with Monte Carlo: For policy evaluation to work for action values, we must assure continual exploration. One ...
user's user avatar
  • 145
3 votes
1 answer
139 views

Why does REINFORCE perform badly at first in Sutton and Barto Figure 13.1?

In Sutton and Barto (PDF, page 265), 2nd edition, Figure 13.1 applies REINFORCE to the "short corridor with switched actions" environment from Example 13.1. The figure looks like this: My ...
LarrySnyder610's user avatar
1 vote
1 answer
95 views

What does "All store and access operations (for S(t) , A(t), and R(t)) can take their index mod n + 1" mean?

It's from the book Introduction to Reinforcement Learning. Second edition, chapter7: n-step Bootstrapping, page 147, n-step Sarsa. I made the algo work, but I still don't understand the phrase. ...
Melanol's user avatar
  • 93
0 votes
1 answer
213 views

Is my derivation of the Bellman equation for $q_{\pi}$ in terms of $p(s'|s,a)$ and $r(s,a)$ correct?

I have done exercise 3.29 from Sutton and Barto and I'd like to check if it's correct. Here's the exercise: Rewrite the Bellman equation for the function $q_{\pi}$ in terms of the three argument ...
user's user avatar
  • 145
0 votes
0 answers
35 views

Rewrite the four Bellman equations for the four value functions $(v_{\pi},v_*,q_{\pi},q_*)$ in terms of $p$ (3.4) and $r$ (3.5) [duplicate]

I have done exercise 3.29 from Sutton and Barto and I'd like to check if it's correct. Here's the exercise: Rewrite the four Bellman equations for the four value functions $(v_{\pi},v_*,q_{\pi},q_*)$ ...
user's user avatar
  • 145