# Questions tagged [sutton-barto]

For questions related to the book "Reinforcement Learning: An Introduction" (by Andrew Barto and Richard S. Sutton).

91 questions
Filter by
Sorted by
Tagged with
25 views

### Understanding SARSA with binary features and function approximation

I'm reading the Sutton and Barto Book on Reinforcement Learning (for oral exam preparation). However, I'm stuck on the algorithm for SARSA($\lambda$) with binary features and linear function ...
1 vote
86 views

### Where is the problem: in batch TD(0) algorithm or in the code to solve AB problem in Sutton-Barto RL book?

Here is the batch TD(0) algorithm: Here is the AB example I want to solve using batch TD(0): And finally here is my Matlab code: % eps1: A 0 B 0 % eps2: B 1 % eps3: B 1 % eps4: B 1 % eps5: B 1 % ...
• 281
1 vote
78 views

### How does off-policy Monte Carlo weighted importance sampling bias converge to zero (Sutton & Barto Section 5.5)

On Section 5.5 (page 105) of Sutton & Barto's "Reinforcement Learning: An Introduction", they discuss the off-policy Monte Carlo method for learning the value function of a target policy ...
• 198
107 views

### Why is $\sum_{s} \eta(s)$ a constant of proportionality in the proof of the policy gradient theorem?

In Sutton and Barto's book (http://incompleteideas.net/book/bookdraft2017nov5.pdf), a proof of the policy gradient theorem is provided on pg. 269 for an episodic case and a start state policy ...
• 51
241 views

### Sutton & Barto: what are parametrized functions?

From "Reinforcement Learning: An introduction (2nd ed.)" by Richard S. Sutton and Andrew G. Barto, on page 59 Instead, the agent would have to maintain $v_\pi$ and $q_\pi$ as parameterized ...
1 vote
46 views

### Why will every action be sampled an infinite number of times?

I am reading the book Reinforcement Learning: An Introduction. Second edition (Richard S. Sutton and Andrew G. Barto). In the k-armed bandit problem using $\varepsilon$-greedy selection method, the ...
1 vote
137 views

• 281
1 vote
126 views

### Unclear step in off-policy (every-visit) MC Control: why do we need the line: $A_t \neq \pi(S_t)$ then exit inner loop?

Could please some expert on reinforcement learning explain the red-box part in the following off-policy MC control? I mean I did not understand what (and why) is done in the step shown as a red-box. I ...
• 179
251 views

### Why does OpenAI's PPO algorithm not follow the discounting method used in Sutton & Barto?

As discussed in this question, the policy gradient algorithms given in Reinforcement Learning: An Introduction use the gradient \begin{align*} \gamma^t \hat A_t \nabla_{\theta} \log \pi(a_t \, | \, ...
• 1,131
65 views

### In Policy Gradient methods, why are actions always chosen from a Gaussian in the literature?

In Sutton's 2020 Reinforcement Learning text (in chapter 13.7 Policy Parameterization for Continuous Actions) it's stated actions [may be] chosen from a normal (Gaussian) distribution. However, I ...
• 121
295 views

### What is the difference between an on-policy distribution and state visitation frequency?

On-policy distribution is defined as follows in Sutton and Barto: On the other hand, state visitation frequency is defined as follows in Trust Region Policy Optimization: \rho_{\pi}(s) = \sum_{t=0}^...
• 349
449 views

### Why would SARSA diverge (but not Expected SARSA or Q-learning)?

In figure 6.3 (shown below) from Reinforcement Learning: An Introduction (second edition) by Sutton and Barto, SARSA is shown to perform worse asymptotically (after 100k episodes) than in the interim (...
73 views

220 views

### If $\gamma \in (0,1)$, what is the on-policy state distribution for episodic tasks?

In Reinforcement Learning: An Introduction, section 9.2 (page 199), Sutton and Barto describe the on-policy distribution in episodic tasks, with $\gamma =1$, as being \begin{equation} \mu(s) = \frac{\...
• 103
104 views

### Is the existence and uniqueness of the state-value function for $\gamma < 1$ theoretical?

Consider the following statement from 4.1 Policy Evaluation of the first edition of Sutton and Barto's book. The existence and uniqueness of $V^{\pi}$ are guaranteed as long as either $\gamma < 1$...
• 3,501
152 views

### How to simplify policy gradient theorem to $E_{\pi}[G_t \frac{\nabla_{\theta}\pi(a|S_t,\theta)}{\pi(a|S_t,\theta)}]$?

In "Introduction to Reinforcement Learning" (Richard Sutton) section 13.3(Reinforce algorithm) they have the following equation: \begin{align} \nabla_{\theta}J &\propto \sum_s \mu(s) \...
• 663
1 vote
76 views

### Why is the update in-place faster than the out-of-place one in dynamic programming?

In Barto and Sutton's book, it's written that we have two types of updates in dynamic programming Update out-of-place Update in-place The update in-place is the faster one. Why is that the case? ...
• 13
306 views

### Suppose every-visit MC was used instead of first-visit MC on blackjack. Would you expect the results to be different?

This is a question from page 94 of Sutton and Barto's RL book 2020. I read in someone's compiled GitHub answers to this book's exercises their answer was: "No because each state in an episode of ...
• 687