Questions tagged [sutton-barto]

For questions related to the book "Reinforcement Learning: An Introduction" (by Andrew Barto and Richard S. Sutton).

65 questions
Filter by
Sorted by
Tagged with
1 vote
19 views

If the probabilities with which each task is selected for you do not change over time, why would it appear as a single stationary k-armed bandit task?

Sutton-Barto (Section 2.9-Associative Search (Contextual Bandits), page 41): As an example, suppose there are several different k-armed bandit tasks, and that on each step you confront one of these ...
• 149
1 vote
29 views

Gradient bandit algorithm: is $\bar{R}_t$ average of all rewards or average of rewards corresponding to $A_t$?

Sutton-Barto (Section 2.8-Gradient Bandit Algorithms, page 37): Question: is $\bar{R}_t$ average of all rewards or average of rewards corresponding to $A_t$?
• 149
10 views

Knowing the futility of discounting in continuing problems, how can we say discounting has no role in control problems with function approximation?

Sutton-Barto (Section 10.4, page 254): Based on the futility of discounting in continuing problems, how can we conclude that discounting has no role to play in control problems with function ...
• 149
1 vote
11 views

Why is the step-size $\alpha_t = 1/t$ not appropriate for non-stationary function approximation?

Sutton-Barto (Section: Selecting Step-Size Parameters Manually, page: 222) The classical choice $\alpha_t = 1/t$, which produces sample averages in tabular MC methods, is not appropriate for TD ...
• 149
14 views

Proof of Convergence of Linear TD(0): Why do we need to show that each row sum plus the corresponding column sum is positive?

Sutton-Barto (page 206): For our key matrix, $D(I − \gamma P)$, the diagonal entries are positive and the off-diagonal entries are negative, so all we have to show is that each row sum plus the ...
• 194
29 views

Unclear paragraph in Sutton-Barto on "Tile Coding"

Sutton-Barto (Tile Coding, page 218): For example, choosing $\alpha = 1/n$, where n is the number of tilings, results in exact one-trial learning. If the example $s\to v$ is trained on, then whatever ...
• 194
27 views

How state 1 has a 0.5 chance of terminating on the left, and state 950 has a 0.25 chance of terminating on the right?

Sutton-Barto's RL book (page 203) Example 9.1: State Aggregation on the 1000-state Random Walk: Consider a 1000-state version of the random walk task (Examples 6.2 and 7.1 on pages 125 and 144). The ...
• 194
55 views

How is state aggregation defined mathematically here? [duplicate]

Sutton-Barto's RL book (page 203): State aggregation is a simple form of generalizing function approximation in which states are grouped together, with one estimated value (one component of the ...
• 194
9 views

What is meant by "weighting the update of each according to the on-policy distribution"?

In Sutton-Barto's RL book in Section 8.6 "Trajectory Sampling" (page 175), they say: If one had an explicit representation of the on-policy distribution, then one could sweep through all ...
• 194
1 vote
36 views

Without planning, why does each episode only add one additional step to the policy?

In Sutton & Barto's RL book at page 165 for Example 8.1, they say: Figure 8.3 shows why the planning agents found the solution so much faster than the nonplanning agent. Shown are the policies ...
• 194
1 vote
47 views

37 views

How to prove importance sampling ratio is uncorrelated with action-value (or state-value) estimate?

In Sutton & Barto (2nd edition), the following is mentioned on page 150 (p. 172 of the pdf), section 7.4: the importance sampling ratio has expected value one (Section 5.9) and is uncorrelated ...
• 339