Share Your Experience: Take the 2024 Developer Survey

# Questions tagged [sutton-barto]

For questions related to the book "Reinforcement Learning: An Introduction" (by Andrew Barto and Richard S. Sutton).

123 questions
Filter by
Sorted by
Tagged with
10 views

### Confusing convention in Sutto-Barto on Monte Carlo Tree Search: is a leaf node a state leaf node or state-action leaf node?

Figure 8.10: Monte Carlo Tree Search. When the environment changes to a new state, MCTS executes as many iterations as possible before an action needs to be selected, incrementally building a tree ...
• 813
19 views

### Unclear line in prioritized sweeping algorithm

Could someone explain the red line (especially, the meaning of the difference) in prioritized sweeping algorithm below? Sutton-Barto, page 170:
• 813
8 views

### For simulated experience, Rollout algorithms = classical MC control?

Rollout algorithms are decision-time planning algorithms based on Monte Carlo control applied to simulated trajectories (using a model) that all begin at the current state. Does that mean for ...
• 813
33 views

### What are the update equations for Double Expected Sarsa with an $\epsilon$-greedy target policy?

This is question 6.13 in Sutton-Barto,page 136. What are the update equations for Double Expected Sarsa with an $\epsilon$-greedy target policy? The answer is given as follows: Let $Q_1$ and $Q_2$ ...
• 813
39 views

### Confusing statement in Sutton-Barto on trajectory sampling

Suuton-Barto, page 176: experiment to assess the effect empirically. To isolate the e↵ect of the update distribution, we used entirely one-step expected tabular updates, as defined by (8.1). In the ...
• 813
27 views

### Confusing statement in Sutton-Barto on expected versus sample updates

Sutton-Barto, page 174. b successor states are equally likely and in which the error in the initial estimate is 1. The values at the next states are assumed correct, so the expected update reduces ...
• 813
65 views

### Why is dynamic programming an example of planning?

Sutton-Barto, page 160, towards bottom: Why is dynamic programming an example of planning? There is no simulation in dynamic programming.
• 813
12 views

### Is this a typo in n-step tree backup section in Sutton-Barto?

Sutton-Barto, page 153. Should not it be $t<T-n$ in Eq.16? The reason is we have $t<T-1$ and $t<T-2$ for the 1 and 2 step returns, respectively.
• 813
22 views

### Confusing points in Dyna-Q in Sutton-Barto about model, simulated experience and model predictions

Sutton-Barto, page 164: In the pseudocode algorithm for Dyna-Q in the box below, Model(s, a) denotes the contents of the model (predicted next state and reward) for state–action pair (s, a). Direct ...
• 813
19 views

### Unclear points in Dyna Maze example in Sutton-Barto

Sutton-Barto, page 164: The main part of Figure 8.2 shows average learning curves from an experiment in which Dyna-Q agents were applied to the maze task. The initial action values were zero, the ...
• 813
17 views

### Confusing point point in Dyna-Q

Sutton-Barto, page 164: In the (f) loop, $S,A, S', R$ are from real experience (Model(S,A)=(R, S') where R and S' are also real expereince). This experience is used in direct RL part in (d). Why is ...
• 813
19 views

### Unclear arrow in general Dyna architecture

Sutton-Barto, page 163: Figure 8.1: The general Dyna Architecture. Real experience, passing back and forth between the environment and the policy, affects policy and value functions in much the same ...
• 813
25 views

### What is the backed-up value in dynamic programming and the corresponding update based on this backed up value?

Sutton-Barto, page 160: Dynamic programming methods clearly fit this structure: they make sweeps through the space of states, generating for each state the distribution of possible transitions. Each ...
• 813
14 views

### Are these typos in n-step tree backup section in Sutton-Barto?

Sutton-Barto, page 153. It seems to me that the "red" underlined parts are typos. 1-) 2-step tree backup return formula is valid for $t<T-2$ but the n-step version which includes $n\ge 2$...
• 813
39 views

### Why no falling off cliff in SARSA for the example in Sutton-Barto?

Sutton-Barto, page 132: The graph to the right shows the performance of the Sarsa and Qlearning methods with "-greedy action selection, " = 0.1. After an initial transient, Q-learning ...
• 813
15 views

### Unclear point in n-step state value estimation

Sutton-Barto, page 143: Here they say: "To make up for that, an equal number of additional updates are made at the end of the episode, after termination and before starting the next episode.&...
• 813
20 views

### In what cases options framework results in faster learning in RL?

If we define options in such a way that any optimal trajectory from starting point to the goal, you never visit a state that belongs to the set of states that are in any of these options. In what ...
32 views

### Sutton-Barto confusing notation for the target and behaviour policy in the expected sarsa

Sutton-Barto, page 134, second paragraph: In these cliff walking results Expected Sarsa was used on-policy, but in general it might use a policy different from the target policy $\pi$ to generate ...
• 813
80 views

60 views

### How does off-policy monte carlo explore and converge? [duplicate]

Premises to question: Behavior Policy: e-greedy (stochastic) Target Policy: greedy (deterministic) Importance Sampling Included In off-policy Monte-Carlo control, the behavior policy chooses actions ...
39 views

### What does it mean for an episode to start in a state-action pair?

In Sutton and Barto on chapter 5 (p.96), they talk about estimating state-action values with Monte Carlo: For policy evaluation to work for action values, we must assure continual exploration. One ...
• 145
139 views

### Why does REINFORCE perform badly at first in Sutton and Barto Figure 13.1?

In Sutton and Barto (PDF, page 265), 2nd edition, Figure 13.1 applies REINFORCE to the "short corridor with switched actions" environment from Example 13.1. The figure looks like this: My ...
1 vote
95 views

### What does "All store and access operations (for S(t) , A(t), and R(t)) can take their index mod n + 1" mean?

It's from the book Introduction to Reinforcement Learning. Second edition, chapter7: n-step Bootstrapping, page 147, n-step Sarsa. I made the algo work, but I still don't understand the phrase. ...
• 93
213 views

### Is my derivation of the Bellman equation for $q_{\pi}$ in terms of $p(s'|s,a)$ and $r(s,a)$ correct?

I have done exercise 3.29 from Sutton and Barto and I'd like to check if it's correct. Here's the exercise: Rewrite the Bellman equation for the function $q_{\pi}$ in terms of the three argument ...
• 145
### Rewrite the four Bellman equations for the four value functions $(v_{\pi},v_*,q_{\pi},q_*)$ in terms of $p$ (3.4) and $r$ (3.5) [duplicate]
I have done exercise 3.29 from Sutton and Barto and I'd like to check if it's correct. Here's the exercise: Rewrite the four Bellman equations for the four value functions $(v_{\pi},v_*,q_{\pi},q_*)$ ...