The Stack Overflow podcast is back! Listen to an interview with our new CEO.

Questions tagged [rl-an-introduction]

For questions related to the book "Reinforcement Learning: An Introduction" (by Andrew Barto and Richard S. Sutton).

Filter by
Sorted by
Tagged with
1
vote
0answers
48 views

Doubt regarding improvement of State Value by n-step returns

Excerpt from Sutton and Barto: The expected value of all $n$-step returns is guaranteed to improve in a certain way over the current value function as an approximation to the true value ...
2
votes
2answers
63 views

How do we get the true value in the prediction objective in reinforcement learning?

In the book "Reinforcement Learning: An Introduction" (2018) Sutton and Barto define the prediction objective ($\overline{VE}$) as follows (page 199): $$\overline{VE}\doteq\sum_{s\epsilon S} \mu(s)[v_{...
1
vote
0answers
44 views

Hashed Tile Coding vs Regular Tile Coding

In the book "Reinforcement Learning: An Introduction" (2018) Sutton and Barto explain at page 221 a form of tile coding using hashing, to reduce memory consumption. I have two questions about that: ...
2
votes
1answer
50 views

Possible inconsistency in the Policy Improvement equation

I came across this formula in Sutton And Barto: RL an Intro (2nd Edition) equation number 4.7 (page number 78). If $\pi$ and $\pi'$ are deterministic policies and $q_\pi(s, \pi'(s)) \geq v_\pi(s)$ ...
4
votes
1answer
126 views

How is the policy gradient calculated in REINFORCE?

Reading Sutton and Barto, I see the following in describing policy gradients: How is the gradient calculated with respect to an action (taken at time t)? I've read implementations of the algorithm, ...
1
vote
1answer
75 views

How can the $\lambda$-return be defined recursively?

The $\lambda$-return is defined as $$G_t^\lambda = (1-\lambda)\sum_{n=1}^\infty \lambda^{n-1}G_{t:t+n}$$ where $$G_{t:t+n} = R_{t+1}+\gamma R_{t+2}+\dots +\gamma^{n-1}R_{t+n} + \gamma^n\hat{v}(S_{t+n})...
3
votes
1answer
74 views

Understanding the n-step off-policy SARSA update

In Sutton & Barto's book (2nd ed) page 149, there is the equation 7.11 I am having a hard time understanding this equation. I would have thought that we should be moving $Q$ towards $G$, where $...
5
votes
2answers
688 views

What is the difference between reinforcement learning and optimal control?

Coming from a process (optimal) control background, I have begun studying the field of deep reinforcement learning. Sutton & Barto (2015) state that particularly important (to the writing of ...
3
votes
2answers
173 views

Difference in continuing and episodic cases in Sutton and Barto - Introduction to RL, exercise 3.5

Excercise 3.5 The equastions in Section 3.1 are for the continuing case and need to be modified (very slightly) to apply to episodic tasks. Show that you know the modifications needed by giving ...
2
votes
1answer
643 views

Expected SARSA vs SARSA in “RL: An Introduction”

Sutton and Barto state in the 2018-version of "Reinforcement Learning: An Introduction" in the context of Expected SARSA (p. 133) the following sentences: Expected SARSA is more complex ...
1
vote
1answer
78 views

How do I apply the value iteration algorithm when there are two goal states?

I am working through the famous RL textbook by Sutton & Barto. Currently, I am on the value iteration chapter. To gain better understanding, I coded up a small example, inspired by this article. ...
1
vote
1answer
138 views

What are the value functions used in reinforcement learning?

In reinforcement learning, we often define two functions, the state-value function $$V^\pi(s) = \mathbb{E}_{\pi} \left[\sum_{k=0}^{\infty} \gamma^{k}R_{t+k+1} \Bigg| S_t=s \right]$$ and the state-...
3
votes
1answer
144 views

How can the importance-sampling ratio be different than zero?

In the book "Reinforcement Learning: An Introduction" (2018) Sutton and Barto define at page 102 the importance-sampling-ration as follows: $$\rho _{t:T-1}=\prod_{k=t}^{T-1}\frac{\pi(A_k|S_k)}{b(A_k|...
1
vote
1answer
51 views

What is the meaning of Model(s, a) in the prioritized sweeping algorithm?

I'm reading the book "Reinforcement Learning: An Introduction" (by Andrew Barto and Richard S. Sutton). The authors provide the pseudocode of the prioritized sweeping algorithm, but I do not know ...
6
votes
1answer
340 views

How do we prove the n-step return error reduction property?

In section 7.1 (about the n-step bootstrapping) of the book Reinforcement Learning: An Introduction (2nd edition), by Andrew Barto and Richard S. Sutton, the authors write about what they call the "n-...
2
votes
1answer
101 views

Understanding the notation in the definition of the expected reward

I am new to RL and I am trying to work through the book Reinforcement Learning: An Introduction I (Sutton & Barto, 2018). In chapter 3 on Finite Markov Decision Processes, the authors write the ...
3
votes
1answer
60 views

Is my interpretation of the return correct?

Sutton and Barto 2018 define the discounted return $G_t$ the following way (p 55): Is my interpretation correct? Or should all "1" be in the same column?
10
votes
3answers
727 views

Why does the discount rate in the REINFORCE algorithm appear twice?

I was reading the book Reinforcement Learning: An Introduction by Richard S. Sutton and Andrew G. Barto (complete draft, November 5, 2017). On page 271, the pseudo-code for the episodic Monte-Carlo ...