# Questions tagged [sutton-barto]

For questions related to the book "Reinforcement Learning: An Introduction" (by Andrew Barto and Richard S. Sutton).

71 questions
Filter by
Sorted by
Tagged with
12 views

### $E_{\pi}[R_{t+1}|S_t=s,A_t=a] = E[R_{t+1}|S_t=s,A_t=a]$?

I would like to solve exercise 3.19 from Sutton and Barto. Here's the exercise: Exercise 3.19 The value of an action, q⇡(s, a), depends on the expected next reward and the expected sum of the ...
• 141
492 views

### In Value Iteration, why can we initialize the value function arbitrarily?

I have not been able to find a good explanation of this, other than statements that the algorithm is guaranteed to converge with arbitrary choices for initial values in each state. Is this something ...
• 73
37 views

### How to perform the back propagation step in Semi-gradient SARSA using a deep neural network?

For the back weight update step, I need to calculate $\nabla\hat{q}(S,A,w)$. My neural network takes in the state vector $S$ and gives out the action values for state $S$ and each action in the action ...
93 views

### Why does the average-reward estimator for continuing tasks use the TD error?

In Sutton and Barto's RL book, section 10.3 describes how to use average reward $r(\pi)$ to define the quality of a policy, re-defining action-value function $q_\pi(s,a)$ and value function $v_\pi(s)$ ...
• 66
82 views

### What should the discount factor for the non-slippery version of the FrozenLake environment be?

I was working with FrozenLake 4x4 from open AI gym. In the slippery case, using a discounting factor of 1, my value iteration implementation was giving a success rate of around 75 percent. It was much ...
1 vote
20 views

### If the probabilities with which each task is selected for you do not change over time, why would it appear as a single stationary k-armed bandit task?

Sutton-Barto (Section 2.9-Associative Search (Contextual Bandits), page 41): As an example, suppose there are several different k-armed bandit tasks, and that on each step you confront one of these ...
• 149
1 vote
34 views

### Gradient bandit algorithm: is $\bar{R}_t$ average of all rewards or average of rewards corresponding to $A_t$?

Sutton-Barto (Section 2.8-Gradient Bandit Algorithms, page 37): Question: is $\bar{R}_t$ average of all rewards or average of rewards corresponding to $A_t$?
• 149
10 views

### Knowing the futility of discounting in continuing problems, how can we say discounting has no role in control problems with function approximation?

Sutton-Barto (Section 10.4, page 254): Based on the futility of discounting in continuing problems, how can we conclude that discounting has no role to play in control problems with function ...
• 149
1 vote
13 views

### Why is the step-size $\alpha_t = 1/t$ not appropriate for non-stationary function approximation?

Sutton-Barto (Section: Selecting Step-Size Parameters Manually, page: 222) The classical choice $\alpha_t = 1/t$, which produces sample averages in tabular MC methods, is not appropriate for TD ...
• 149
23 views

### Proof of Convergence of Linear TD(0): Why do we need to show that each row sum plus the corresponding column sum is positive?

Sutton-Barto (page 206): For our key matrix, $D(I − \gamma P)$, the diagonal entries are positive and the off-diagonal entries are negative, so all we have to show is that each row sum plus the ...
• 194
31 views

### Unclear paragraph in Sutton-Barto on "Tile Coding"

Sutton-Barto (Tile Coding, page 218): For example, choosing $\alpha = 1/n$, where n is the number of tilings, results in exact one-trial learning. If the example $s\to v$ is trained on, then whatever ...
• 194
28 views

### How state 1 has a 0.5 chance of terminating on the left, and state 950 has a 0.25 chance of terminating on the right?

Sutton-Barto's RL book (page 203) Example 9.1: State Aggregation on the 1000-state Random Walk: Consider a 1000-state version of the random walk task (Examples 6.2 and 7.1 on pages 125 and 144). The ...
• 194
58 views

### How is state aggregation defined mathematically here? [duplicate]

Sutton-Barto's RL book (page 203): State aggregation is a simple form of generalizing function approximation in which states are grouped together, with one estimated value (one component of the ...
• 194
10 views

### What is meant by "weighting the update of each according to the on-policy distribution"?

In Sutton-Barto's RL book in Section 8.6 "Trajectory Sampling" (page 175), they say: If one had an explicit representation of the on-policy distribution, then one could sweep through all ...
• 194
1 vote
36 views

### Without planning, why does each episode only add one additional step to the policy?

In Sutton & Barto's RL book at page 165 for Example 8.1, they say: Figure 8.3 shows why the planning agents found the solution so much faster than the nonplanning agent. Shown are the policies ...
• 194
1 vote
48 views

38 views

### How to prove importance sampling ratio is uncorrelated with action-value (or state-value) estimate?

In Sutton & Barto (2nd edition), the following is mentioned on page 150 (p. 172 of the pdf), section 7.4: the importance sampling ratio has expected value one (Section 5.9) and is uncorrelated ...
• 339