Questions tagged [sutton-barto]
For questions related to the book "Reinforcement Learning: An Introduction" (by Andrew Barto and Richard S. Sutton).
71
questions
3
votes
1
answer
12
views
$E_{\pi}[R_{t+1}|S_t=s,A_t=a] = E[R_{t+1}|S_t=s,A_t=a]$?
I would like to solve exercise 3.19 from Sutton and Barto. Here's the exercise:
Exercise 3.19 The value of an action, q⇡(s, a), depends on the expected next reward and
the expected sum of the ...
- 141
7
votes
2
answers
492
views
In Value Iteration, why can we initialize the value function arbitrarily?
I have not been able to find a good explanation of this, other than statements that the algorithm is guaranteed to converge with arbitrary choices for initial values in each state. Is this something ...
- 73
0
votes
1
answer
37
views
How to perform the back propagation step in Semi-gradient SARSA using a deep neural network?
For the back weight update step, I need to calculate $\nabla\hat{q}(S,A,w)$. My neural network takes in the state vector $S$ and gives out the action values for state $S$ and each action in the action ...
2
votes
1
answer
93
views
Why does the average-reward estimator for continuing tasks use the TD error?
In Sutton and Barto's RL book, section 10.3 describes how to use average reward $r(\pi)$ to define the quality of a policy, re-defining action-value function $q_\pi(s,a)$ and value function $v_\pi(s)$ ...
- 66
4
votes
1
answer
82
views
What should the discount factor for the non-slippery version of the FrozenLake environment be?
I was working with FrozenLake 4x4 from open AI gym. In the slippery case, using a discounting factor of 1, my value iteration implementation was giving a success rate of around 75 percent. It was much ...
1
vote
1
answer
20
views
If the probabilities with which each task is selected for you do not change over time, why would it appear as a single stationary k-armed bandit task?
Sutton-Barto (Section 2.9-Associative Search (Contextual Bandits), page 41):
As an example, suppose there are several different k-armed bandit tasks, and that on
each step you confront one of these ...
- 149
1
vote
1
answer
34
views
Gradient bandit algorithm: is $\bar{R}_t$ average of all rewards or average of rewards corresponding to $A_t$?
Sutton-Barto (Section 2.8-Gradient Bandit Algorithms, page 37):
Question: is $\bar{R}_t$ average of all rewards or average of rewards corresponding to $A_t$?
- 149
0
votes
0
answers
10
views
Knowing the futility of discounting in continuing problems, how can we say discounting has no role in control problems with function approximation?
Sutton-Barto (Section 10.4, page 254):
Based on the futility of discounting in continuing problems, how can we conclude that discounting has no role to play in control problems with function ...
- 149
1
vote
0
answers
13
views
Why is the step-size $\alpha_t = 1/t$ not appropriate for non-stationary function approximation?
Sutton-Barto (Section: Selecting Step-Size Parameters Manually, page: 222)
The classical choice $\alpha_t = 1/t$, which produces sample averages in tabular MC methods, is not appropriate for TD ...
- 149
0
votes
0
answers
23
views
Proof of Convergence of Linear TD(0): Why do we need to show that each row sum plus the corresponding column sum is positive?
Sutton-Barto (page 206):
For our key matrix, $D(I − \gamma P)$, the diagonal entries are positive
and the off-diagonal entries are negative, so all we have to show is that each row
sum plus the ...
- 194
0
votes
1
answer
31
views
Unclear paragraph in Sutton-Barto on "Tile Coding"
Sutton-Barto (Tile Coding, page 218):
For example, choosing $\alpha = 1/n$, where n is the number
of tilings, results in exact one-trial learning. If the example $s\to v$ is trained on, then
whatever ...
- 194
0
votes
1
answer
28
views
How state 1 has a 0.5 chance of terminating on the left, and state 950 has a 0.25 chance of terminating on the right?
Sutton-Barto's RL book (page 203)
Example 9.1: State Aggregation on the 1000-state Random Walk: Consider a
1000-state version of the random walk task (Examples 6.2 and 7.1 on pages 125 and
144). The ...
- 194
0
votes
1
answer
58
views
How is state aggregation defined mathematically here? [duplicate]
Sutton-Barto's RL book (page 203):
State aggregation is a simple form of generalizing function approximation in which
states are grouped together, with one estimated value (one component of the ...
- 194
0
votes
0
answers
10
views
What is meant by "weighting the update of each according to the on-policy distribution"?
In Sutton-Barto's RL book in Section 8.6 "Trajectory Sampling" (page 175), they say:
If one had an explicit representation
of the on-policy distribution, then one could sweep through all ...
- 194
1
vote
1
answer
36
views
Without planning, why does each episode only add one additional step to the policy?
In Sutton & Barto's RL book at page 165 for Example 8.1, they say:
Figure 8.3 shows why the planning agents found the solution so much faster than the nonplanning agent. Shown are the policies ...
- 194
1
vote
0
answers
48
views
Why does importance sampling ratio start and end one step later in off-policy SARSA given in Sutton-Barto's RL book?
In Sutton & Barto's RL book (page 149) they say:
Sarsa update can be completely replaced by a simple off-policy form
$Q_{t+n}(S_t,A_t)=Q_{t+n−1}(S_t,A_t) + \rho_{t+1:t+n} [G_{t:t+n} − Q_{t+n−1}(...
- 194
0
votes
1
answer
38
views
Why $ t=τ+n-1$ instead of $t=τ+n$ in n-step TD?
If $\tau$ is the time, whose state’s estimate is being updated, and $t$ is the current time, then, in n-step TD method, we have $t=\tau+n$ (because we have to wait n-steps, before we can update). ...
- 194
2
votes
1
answer
50
views
Why do we have $t$ as subscript in $V$ instead of $t+1$ in the expression of $G_{t:t+1}$?
In one-step TD updates, the target is the first reward plus the discounted estimated value of the next state, which we call the one-step return (page 143 of Sutton & Barto):
$$
G_{t:t+1} \...
- 194
0
votes
2
answers
94
views
Why does OpenAI's PPO algorithm not follow the discounting method used in Sutton & Barto?
As discussed in this question, the policy gradient algorithms given in Reinforcement Learning: An Introduction use the gradient
\begin{align*}
\gamma^t \hat A_t \nabla_{\theta} \log \pi(a_t \, | \, ...
- 876
2
votes
0
answers
46
views
In Policy Gradient methods, why are actions always chosen from a Gaussian in the literature?
In Sutton's 2020 Reinforcement Learning text (in chapter 13.7 Policy Parameterization for Continuous Actions) it's stated
actions [may be] chosen from a normal (Gaussian) distribution.
However, I ...
- 121
3
votes
0
answers
72
views
What is the difference between an on-policy distribution and state visitation frequency?
On-policy distribution is defined as follows in Sutton and Barto:
On the other hand, state visitation frequency is defined as follows in Trust Region Policy Optimization:
$$\rho_{\pi}(s) = \sum_{t=0}^...
- 339
1
vote
1
answer
188
views
Why would SARSA diverge (but not Expected SARSA or Q-learning)?
In figure 6.3 (shown below) from Reinforcement Learning: An Introduction (second edition) by Sutton and Barto, SARSA is shown to perform worse asymptotically (after 100k episodes) than in the interim (...
- 111
0
votes
1
answer
40
views
Policies for which the policy improvement theorem holds
According to Reinforcement Learning (2nd Edition) by Sutton and Barto, the policy improvement theorem states that for any pair of deterministic policies $\pi'$ and $\pi$, if $q_\pi(s,\pi'(s)) \geq v_\...
2
votes
1
answer
38
views
How to prove importance sampling ratio is uncorrelated with action-value (or state-value) estimate?
In Sutton & Barto (2nd edition), the following is mentioned on page 150 (p. 172 of the pdf), section 7.4:
the importance sampling ratio has expected value one (Section 5.9) and is uncorrelated ...
- 339
0
votes
0
answers
17
views
Why does one-step TD strengthen only the last action of the sequence of actions that led to the high reward, while n-step TD the last n actions?
In the caption of figure 7.4 (p. 147) of Sutton & Barto's book (2nd edition), it's written
The one-step method strengthens only the last action of the sequence of actions that led to the high ...
- 339
2
votes
1
answer
101
views
In off-policy MC control algorithm by Sutton & Barto, why do we perform a last update when sample action is inconsistent with target policy?
I have a question about the $W$ term in the off-policy MC control algorithm on Page 111 of Sutton & Barto. I have also included it in the figure below.
My question: shouldn't the check $A_{t} = \...
- 123
6
votes
1
answer
136
views
If $\gamma \in (0,1)$, what is the on-policy state distribution for episodic tasks?
In Reinforcement Learning: An Introduction, section 9.2 (page 199), Sutton and Barto describe the on-policy distribution in episodic tasks, with $\gamma =1$, as being
\begin{equation}
\mu(s) = \frac{\...
- 93
2
votes
1
answer
71
views
Is the existence and uniqueness of the state-value function for $\gamma < 1$ theoretical?
Consider the following statement from 4.1 Policy Evaluation of the first edition of Sutton and Barto's book.
The existence and uniqueness of $V^{\pi}$ are guaranteed as long as
either $\gamma < 1$...
- 3,099
0
votes
1
answer
116
views
How to simplify policy gradient theorem to $E_{\pi}[G_t \frac{\nabla_{\theta}\pi(a|S_t,\theta)}{\pi(a|S_t,\theta)}]$?
In "Introduction to Reinforcement Learning" (Richard Sutton) section 13.3(Reinforce algorithm) they have the following equation:
\begin{align}
\nabla_{\theta}J &\propto \sum_s \mu(s) \...
- 665
1
vote
1
answer
54
views
Why is the update in-place faster than the out-of-place one in dynamic programming?
In Barto and Sutton's book, it's written that we have two types of updates in dynamic programming
Update out-of-place
Update in-place
The update in-place is the faster one. Why is that the case?
...
- 13
0
votes
0
answers
87
views
Suppose every-visit MC was used instead of first-visit MC on blackjack. Would you expect the results to be different?
This is a question from page 94 of Sutton and Barto's RL book 2020.
I read in someone's compiled GitHub answers to this book's exercises their answer was: "No because each state in an episode of ...
- 597
4
votes
1
answer
209
views
Why do we have two similar action selection strategies for UCB1?
In the literature, there are at least two action selection strategies associated with the UCB1's action selection strategy/policy. For example, in the paper Algorithms for the multi-armed bandit ...
- 33.8k
2
votes
1
answer
62
views
Why is the fraction of time spent in state $s$, $\mu(s)$, not in the update rule of the parameters?
I am reading "Reinforcement Learning: An Introduction (2nd edition)" authored by Sutton and Barto. In Section 9, On-policy prediction with approximation, it first gives the mean squared ...
- 123
3
votes
1
answer
96
views
How do we derive the expression for average reward setting in continuing tasks?
In the average reward setting we have:
$$r(\pi)\doteq \lim_{h\rightarrow\infty}\frac{1}{h}\sum_{t=1}^{h}\mathbb{E}[R_{t}|S_0,A_{0:t-1}\sim\pi]$$
$$r(\pi)\doteq \lim_{t\rightarrow\infty}\mathbb{E}[R_{t}...
- 147
4
votes
2
answers
108
views
Why do all states appear identical under the function approximation in the Short Corridor task?
This is the Short Corridor problem taken from the Sutton & Barto book. Here it's written:
The problem is difficult because all the states appear identical under the function approximation
But ...
- 147
3
votes
1
answer
127
views
What is wrong with equation 7.3 in Sutton & Barto's book?
Equation 7.3 of Sutton Barto book:
$$\text{Equation: } max_s|\mathbb{E}_\pi[G_{t:t+n}|S_t = s] - v_\pi| \le \gamma^nmax_s|V_{t+n-1}(s) - v_\pi(s)| $$
$$\text{where }G_{t:t+n} = R_{t+1} + \gamma R_{t+2}...
- 147
1
vote
1
answer
328
views
Value Iteration failing to converge to optimal value function in Sutton-Barto's Gambler problem
In Example 4.3:Gambler's Problem of Sutton and Barto's book whose code is given here.
In this code the value function array is initialized as np.zeros(states) where ...
- 147
2
votes
2
answers
164
views
How exactly is $Pr(s \rightarrow x, k, \pi)$ deduced by "unrolling", in the proof of the policy gradient theorem?
In the proof of the policy gradient theorem in the RL book of Sutton and Barto (that I shamelessly paste here):
there is the "unrolling" step that is supposed to be immediately clear
With ...
- 362
3
votes
1
answer
232
views
How to express $v_\pi(s)$ in terms of $q_\pi(s,a)$?
This is exercise 3.18 in Sutton and Barto's book.
The task is to express $v_\pi(s)$ using $q_\pi(s,a)$.
Looking at the diagram above, the value of $q_\pi(s,a)$ at $s$ for each $a \in A$ we take gives ...
- 362
2
votes
1
answer
146
views
Why is there an inconsistency between my calculations of Policy Iteration and this Sutton & Barto's diagram?
In equation 4.9 of Sutton and Barto's book on page 79, we have (for the policy iteration algorithm):
$$\pi'(s) = arg \max_{a}\sum_{s',r}p(s',r|s,a)[r+\gamma v_{\pi}(s')]$$
where $\pi$ is the previous ...
- 147
5
votes
1
answer
551
views
If the current state is $S_t$ and the actions are chosen according to $\pi$, what is the expectation of $R_{t+1}$ in terms of $\pi$ and $p$?
I'm trying to solve exercise 3.11 from the book Sutton and Barto's book (2nd edition)
Exercise 3.11 If the current state is $S_t$ , and actions are selected according to a stochastic policy $\pi$, ...
- 362
3
votes
1
answer
152
views
Doubt regarding the proof of convergence of $\epsilon$ soft policies without exploring starts
In page 125 of Sutton and Barto (second last paragraph) the proof for equality of $v_{\pi}$ and $v_*$ for $\epsilon$ soft policies is given. But I could not understand the statement explaining the ...
user9947
4
votes
2
answers
502
views
Why does the definition of the reward function $r(s, a, s')$ involve the term $p(s' \mid s, a)$?
Sutton and Barto define the state–action–next-state reward function, $r(s, a, s')$, as follows (equation 3.6, p. 49)
$$
r(s, a, s^{\prime}) \doteq \mathbb{E}\left[R_{t} \mid S_{t-1}=s, A_{t-1}=a, S_{t}...
- 137
2
votes
1
answer
172
views
What does the figure "Blackjack Value Function..." from Sutton represent?
I came across this graph in David Silver's youtube lecture and Sutton's book on reinforcement learning.
Can anyone help me understand the graph?
From the graph, for 10000 episodes what i see is ...
- 374
3
votes
2
answers
232
views
On-policy state distribution for episodic tasks on Sutton & Barto, page 199
In Sutton & Barto's "Reinforcement Learning: An Introduction", 2nd edition, page 199, they describe the on-policy distribution for episodic tasks in the following box:
I don't understand how this ...
- 177
12
votes
4
answers
1k
views
Counterexamples to the reward hypothesis
On Sutton and Barto's RL book, the reward hypothesis is stated as
that all of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative ...
- 221
3
votes
1
answer
106
views
Sutton & Barto's notation $V_{t+n}$ in Chapter 7: $n$-step Bootstrapping
Until Chapter 6 of Sutton & Barto's book on Reinforcement Learning, the authors use $V$ for the current estimate of a state value. Equation (6.1), for example, shows:
$$ V(S_t) \leftarrow V(S_t) +...
- 177
4
votes
1
answer
610
views
Should the policy parameters be updated at each time step or at the end of the episode in REINFORCE?
REINFORCE is a Monte Carlo policy gradient algorithm, which updates weights (parameters) of policy network by generating episodes. Here's a pseudo-code from Sutton's book (which is same as the ...
- 163
3
votes
1
answer
128
views
What knowledge is required for understanding the AlphaZero paper?
My goal is to understand AlphaZero paper published by deepmind. I'm beginning my journey trying to get the basic intuition of reinforcement learning from the book by Barto and Sutton.
As per my ...
- 133
4
votes
1
answer
150
views
Why is an average of all returns used to update the value in the first-visit MC control?
In Sutton & Barto's Reinforcement Learning: An Introduction, in page 83 (101 of the pdf), there is a description of first-visit MC control. In the phase where they update $Q(s, a)$, they do an ...
- 747