Questions tagged [sutton-barto]

For questions related to the book "Reinforcement Learning: An Introduction" (by Andrew Barto and Richard S. Sutton).

Filter by
Sorted by
Tagged with
1 vote
1 answer
19 views

If the probabilities with which each task is selected for you do not change over time, why would it appear as a single stationary k-armed bandit task?

Sutton-Barto (Section 2.9-Associative Search (Contextual Bandits), page 41): As an example, suppose there are several different k-armed bandit tasks, and that on each step you confront one of these ...
user avatar
1 vote
1 answer
29 views

Gradient bandit algorithm: is $\bar{R}_t$ average of all rewards or average of rewards corresponding to $A_t$?

Sutton-Barto (Section 2.8-Gradient Bandit Algorithms, page 37): Question: is $\bar{R}_t$ average of all rewards or average of rewards corresponding to $A_t$?
user avatar
0 votes
0 answers
10 views

Knowing the futility of discounting in continuing problems, how can we say discounting has no role in control problems with function approximation?

Sutton-Barto (Section 10.4, page 254): Based on the futility of discounting in continuing problems, how can we conclude that discounting has no role to play in control problems with function ...
user avatar
1 vote
0 answers
11 views

Why is the step-size $\alpha_t = 1/t$ not appropriate for non-stationary function approximation?

Sutton-Barto (Section: Selecting Step-Size Parameters Manually, page: 222) The classical choice $\alpha_t = 1/t$, which produces sample averages in tabular MC methods, is not appropriate for TD ...
user avatar
0 votes
0 answers
14 views

Proof of Convergence of Linear TD(0): Why do we need to show that each row sum plus the corresponding column sum is positive?

Sutton-Barto (page 206): For our key matrix, $D(I − \gamma P)$, the diagonal entries are positive and the off-diagonal entries are negative, so all we have to show is that each row sum plus the ...
user avatar
0 votes
1 answer
29 views

Unclear paragraph in Sutton-Barto on "Tile Coding"

Sutton-Barto (Tile Coding, page 218): For example, choosing $\alpha = 1/n$, where n is the number of tilings, results in exact one-trial learning. If the example $s\to v$ is trained on, then whatever ...
user avatar
0 votes
1 answer
27 views

How state 1 has a 0.5 chance of terminating on the left, and state 950 has a 0.25 chance of terminating on the right?

Sutton-Barto's RL book (page 203) Example 9.1: State Aggregation on the 1000-state Random Walk: Consider a 1000-state version of the random walk task (Examples 6.2 and 7.1 on pages 125 and 144). The ...
user avatar
0 votes
1 answer
55 views

How is state aggregation defined mathematically here? [duplicate]

Sutton-Barto's RL book (page 203): State aggregation is a simple form of generalizing function approximation in which states are grouped together, with one estimated value (one component of the ...
user avatar
0 votes
0 answers
9 views

What is meant by "weighting the update of each according to the on-policy distribution"?

In Sutton-Barto's RL book in Section 8.6 "Trajectory Sampling" (page 175), they say: If one had an explicit representation of the on-policy distribution, then one could sweep through all ...
user avatar
1 vote
1 answer
36 views

Without planning, why does each episode only add one additional step to the policy?

In Sutton & Barto's RL book at page 165 for Example 8.1, they say: Figure 8.3 shows why the planning agents found the solution so much faster than the nonplanning agent. Shown are the policies ...
user avatar
1 vote
0 answers
47 views

Why does importance sampling ratio start and end one step later in off-policy SARSA given in Sutton-Barto's RL book?

In Sutton & Barto's RL book (page 149) they say: Sarsa update can be completely replaced by a simple off-policy form $Q_{t+n}(S_t,A_t)=Q_{t+n−1}(S_t,A_t) + \rho_{t+1:t+n} [G_{t:t+n} − Q_{t+n−1}(...
user avatar
0 votes
1 answer
36 views

Why $ t=τ+n-1$ instead of $t=τ+n$ in n-step TD?

If $\tau$ is the time, whose state’s estimate is being updated, and $t$ is the current time, then, in n-step TD method, we have $t=\tau+n$ (because we have to wait n-steps, before we can update). ...
user avatar
2 votes
1 answer
48 views

Why do we have $t$ as subscript in $V$ instead of $t+1$ in the expression of $G_{t:t+1}$?

In one-step TD updates, the target is the first reward plus the discounted estimated value of the next state, which we call the one-step return (page 143 of Sutton & Barto): $$ G_{t:t+1} \...
user avatar
0 votes
2 answers
81 views

Why does OpenAI's PPO algorithm not follow the discounting method used in Sutton & Barto?

As discussed in this question, the policy gradient algorithms given in Reinforcement Learning: An Introduction use the gradient \begin{align*} \gamma^t \hat A_t \nabla_{\theta} \log \pi(a_t \, | \, ...
user avatar
  • 746
2 votes
0 answers
46 views

In Policy Gradient methods, why are actions always chosen from a Gaussian in the literature?

In Sutton's 2020 Reinforcement Learning text (in chapter 13.7 Policy Parameterization for Continuous Actions) it's stated actions [may be] chosen from a normal (Gaussian) distribution. However, I ...
user avatar
3 votes
0 answers
60 views

What is the difference between an on-policy distribution and state visitation frequency?

On-policy distribution is defined as follows in Sutton and Barto: On the other hand, state visitation frequency is defined as follows in Trust Region Policy Optimization: $$\rho_{\pi}(s) = \sum_{t=0}^...
user avatar
1 vote
1 answer
151 views

Why would SARSA diverge (but not Expected SARSA or Q-learning)?

In figure 6.3 (shown below) from Reinforcement Learning: An Introduction (second edition) by Sutton and Barto, SARSA is shown to perform worse asymptotically (after 100k episodes) than in the interim (...
user avatar
0 votes
1 answer
39 views

Policies for which the policy improvement theorem holds

According to Reinforcement Learning (2nd Edition) by Sutton and Barto, the policy improvement theorem states that for any pair of deterministic policies $\pi'$ and $\pi$, if $q_\pi(s,\pi'(s)) \geq v_\...
user avatar
2 votes
1 answer
37 views

How to prove importance sampling ratio is uncorrelated with action-value (or state-value) estimate?

In Sutton & Barto (2nd edition), the following is mentioned on page 150 (p. 172 of the pdf), section 7.4: the importance sampling ratio has expected value one (Section 5.9) and is uncorrelated ...
user avatar
0 votes
0 answers
17 views

Why does one-step TD strengthen only the last action of the sequence of actions that led to the high reward, while n-step TD the last n actions?

In the caption of figure 7.4 (p. 147) of Sutton & Barto's book (2nd edition), it's written The one-step method strengthens only the last action of the sequence of actions that led to the high ...
user avatar
2 votes
1 answer
90 views

In off-policy MC control algorithm by Sutton & Barto, why do we perform a last update when sample action is inconsistent with target policy?

I have a question about the $W$ term in the off-policy MC control algorithm on Page 111 of Sutton & Barto. I have also included it in the figure below. My question: shouldn't the check $A_{t} = \...
user avatar
6 votes
1 answer
131 views

If $\gamma \in (0,1)$, what is the on-policy state distribution for episodic tasks?

In Reinforcement Learning: An Introduction, section 9.2 (page 199), Sutton and Barto describe the on-policy distribution in episodic tasks, with $\gamma =1$, as being \begin{equation} \mu(s) = \frac{\...
user avatar
2 votes
1 answer
68 views

Is the existence and uniqueness of the state-value function for $\gamma < 1$ theoretical?

Consider the following statement from 4.1 Policy Evaluation of the first edition of Sutton and Barto's book. The existence and uniqueness of $V^{\pi}$ are guaranteed as long as either $\gamma < 1$...
user avatar
  • 2,977
0 votes
1 answer
115 views

How to simplify policy gradient theorem to $E_{\pi}[G_t \frac{\nabla_{\theta}\pi(a|S_t,\theta)}{\pi(a|S_t,\theta)}]$?

In "Introduction to Reinforcement Learning" (Richard Sutton) section 13.3(Reinforce algorithm) they have the following equation: \begin{align} \nabla_{\theta}J &\propto \sum_s \mu(s) \...
user avatar
1 vote
1 answer
53 views

Why is the update in-place faster than the out-of-place one in dynamic programming?

In Barto and Sutton's book, it's written that we have two types of updates in dynamic programming Update out-of-place Update in-place The update in-place is the faster one. Why is that the case? ...
user avatar
0 votes
0 answers
77 views

Suppose every-visit MC was used instead of first-visit MC on blackjack. Would you expect the results to be different?

This is a question from page 94 of Sutton and Barto's RL book 2020. I read in someone's compiled GitHub answers to this book's exercises their answer was: "No because each state in an episode of ...
user avatar
4 votes
1 answer
189 views

Why do we have two similar action selection strategies for UCB1?

In the literature, there are at least two action selection strategies associated with the UCB1's action selection strategy/policy. For example, in the paper Algorithms for the multi-armed bandit ...
user avatar
  • 33k
2 votes
1 answer
61 views

Why is the fraction of time spent in state $s$, $\mu(s)$, not in the update rule of the parameters?

I am reading "Reinforcement Learning: An Introduction (2nd edition)" authored by Sutton and Barto. In Section 9, On-policy prediction with approximation, it first gives the mean squared ...
user avatar
  • 123
3 votes
1 answer
88 views

How do we derive the expression for average reward setting in continuing tasks?

In the average reward setting we have: $$r(\pi)\doteq \lim_{h\rightarrow\infty}\frac{1}{h}\sum_{t=1}^{h}\mathbb{E}[R_{t}|S_0,A_{0:t-1}\sim\pi]$$ $$r(\pi)\doteq \lim_{t\rightarrow\infty}\mathbb{E}[R_{t}...
user avatar
4 votes
2 answers
99 views

Why do all states appear identical under the function approximation in the Short Corridor task?

This is the Short Corridor problem taken from the Sutton & Barto book. Here it's written: The problem is difficult because all the states appear identical under the function approximation But ...
user avatar
3 votes
1 answer
125 views

What is wrong with equation 7.3 in Sutton & Barto's book?

Equation 7.3 of Sutton Barto book: $$\text{Equation: } max_s|\mathbb{E}_\pi[G_{t:t+n}|S_t = s] - v_\pi| \le \gamma^nmax_s|V_{t+n-1}(s) - v_\pi(s)| $$ $$\text{where }G_{t:t+n} = R_{t+1} + \gamma R_{t+2}...
user avatar
1 vote
1 answer
304 views

Value Iteration failing to converge to optimal value function in Sutton-Barto's Gambler problem

In Example 4.3:Gambler's Problem of Sutton and Barto's book whose code is given here. In this code the value function array is initialized as np.zeros(states) where ...
user avatar
2 votes
2 answers
148 views

How exactly is $Pr(s \rightarrow x, k, \pi)$ deduced by "unrolling", in the proof of the policy gradient theorem?

In the proof of the policy gradient theorem in the RL book of Sutton and Barto (that I shamelessly paste here): there is the "unrolling" step that is supposed to be immediately clear With ...
user avatar
  • 352
3 votes
1 answer
211 views

How to express $v_\pi(s)$ in terms of $q_\pi(s,a)$?

This is exercise 3.18 in Sutton and Barto's book. The task is to express $v_\pi(s)$ using $q_\pi(s,a)$. Looking at the diagram above, the value of $q_\pi(s,a)$ at $s$ for each $a \in A$ we take gives ...
user avatar
  • 352
2 votes
1 answer
146 views

Why is there an inconsistency between my calculations of Policy Iteration and this Sutton & Barto's diagram?

In equation 4.9 of Sutton and Barto's book on page 79, we have (for the policy iteration algorithm): $$\pi'(s) = arg \max_{a}\sum_{s',r}p(s',r|s,a)[r+\gamma v_{\pi}(s')]$$ where $\pi$ is the previous ...
user avatar
4 votes
1 answer
540 views

If the current state is $S_t$ and the actions are chosen according to $\pi$, what is the expectation of $R_{t+1}$ in terms of $\pi$ and $p$?

I'm trying to solve exercise 3.11 from the book Sutton and Barto's book (2nd edition) Exercise 3.11 If the current state is $S_t$ , and actions are selected according to a stochastic policy $\pi$, ...
user avatar
  • 352
3 votes
1 answer
148 views

Doubt regarding the proof of convergence of $\epsilon$ soft policies without exploring starts

In page 125 of Sutton and Barto (second last paragraph) the proof for equality of $v_{\pi}$ and $v_*$ for $\epsilon$ soft policies is given. But I could not understand the statement explaining the ...
user avatar
4 votes
2 answers
483 views

Why does the definition of the reward function $r(s, a, s')$ involve the term $p(s' \mid s, a)$?

Sutton and Barto define the state–action–next-state reward function, $r(s, a, s')$, as follows (equation 3.6, p. 49) $$ r(s, a, s^{\prime}) \doteq \mathbb{E}\left[R_{t} \mid S_{t-1}=s, A_{t-1}=a, S_{t}...
user avatar
2 votes
1 answer
168 views

What does the figure "Blackjack Value Function..." from Sutton represent?

I came across this graph in David Silver's youtube lecture and Sutton's book on reinforcement learning. Can anyone help me understand the graph? From the graph, for 10000 episodes what i see is ...
user avatar
3 votes
2 answers
214 views

On-policy state distribution for episodic tasks on Sutton & Barto, page 199

In Sutton & Barto's "Reinforcement Learning: An Introduction", 2nd edition, page 199, they describe the on-policy distribution for episodic tasks in the following box: I don't understand how this ...
user avatar
12 votes
4 answers
1k views

Counterexamples to the reward hypothesis

On Sutton and Barto's RL book, the reward hypothesis is stated as that all of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative ...
user avatar
  • 221
3 votes
1 answer
100 views

Sutton & Barto's notation $V_{t+n}$ in Chapter 7: $n$-step Bootstrapping

Until Chapter 6 of Sutton & Barto's book on Reinforcement Learning, the authors use $V$ for the current estimate of a state value. Equation (6.1), for example, shows: $$ V(S_t) \leftarrow V(S_t) +...
user avatar
3 votes
1 answer
125 views

What knowledge is required for understanding the AlphaZero paper?

My goal is to understand AlphaZero paper published by deepmind. I'm beginning my journey trying to get the basic intuition of reinforcement learning from the book by Barto and Sutton. As per my ...
user avatar
4 votes
1 answer
146 views

Why is an average of all returns used to update the value in the first-visit MC control?

In Sutton & Barto's Reinforcement Learning: An Introduction, in page 83 (101 of the pdf), there is a description of first-visit MC control. In the phase where they update $Q(s, a)$, they do an ...
user avatar
2 votes
2 answers
136 views

How do we get the true value in the prediction objective in reinforcement learning?

In the book "Reinforcement Learning: An Introduction" (2018) Sutton and Barto define the prediction objective ($\overline{VE}$) as follows (page 199): $$\overline{VE}\doteq\sum_{s\epsilon S} \mu(s)[v_{...
user avatar
  • 291
4 votes
0 answers
218 views

When using hashing in tile coding, why are memory requirements reduced and there is only a little loss of performance?

In the book "Reinforcement Learning: An Introduction" (2018) Sutton and Barto explain, on page 221, a form of tile coding using hashing, to reduce memory consumption. I have two questions ...
user avatar
  • 291
2 votes
1 answer
87 views

Possible inconsistency in the Policy Improvement equation

I came across this formula in Sutton And Barto: RL an Intro (2nd Edition) equation number 4.7 (page number 78). If $\pi$ and $\pi'$ are deterministic policies and $q_\pi(s, \pi'(s)) \geq v_\pi(s)$ ...
user avatar
8 votes
1 answer
900 views

How is the policy gradient calculated in REINFORCE?

Reading Sutton and Barto, I see the following in describing policy gradients: How is the gradient calculated with respect to an action (taken at time t)? I've read implementations of the algorithm, ...
user avatar
  • 499
1 vote
1 answer
784 views

How can the $\lambda$-return be defined recursively?

The $\lambda$-return is defined as $$G_t^\lambda = (1-\lambda)\sum_{n=1}^\infty \lambda^{n-1}G_{t:t+n}$$ where $$G_{t:t+n} = R_{t+1}+\gamma R_{t+2}+\dots +\gamma^{n-1}R_{t+n} + \gamma^n\hat{v}(S_{t+n})...
user avatar
5 votes
1 answer
283 views

Understanding the n-step off-policy SARSA update

In Sutton & Barto's book (2nd ed) page 149, there is the equation 7.11 I am having a hard time understanding this equation. I would have thought that we should be moving $Q$ towards $G$, where $...
user avatar