Planned maintenance impacting Stack Overflow and all Stack Exchange sites is scheduled for Monday, September 16, 2024, 5:00 PM-10:00 PM EDT (Monday, September 16, 21:00 UTC- Tuesday, September 17, 2:00 UTC). The email/password authentication method will be unavailable for logging in and registering. Read more here

# Tag Info

Accepted

### Where does the TD formula for tic-tac-toe in Sutton & Barto come from?

The equation (1) you mentioned is a simplified form of the temporal difference TD(0) update rule, specifically for the case of episodic tasks where there's no discounting $\gamma$ and the only reward ...
• 3,302
Accepted

### Unclear paragraph in Sutton-Barto on exploration/exploitation relating to bandit like decision tasks

The quoted text is from the end of a paragraph that is explaining some aspects of bandit algorithms, and looking forward to how knowledge of them is applicable in a reinforcement learning context. ...
• 32.9k

### Why is dynamic programming an example of planning?

Dynamic programming is a algorithm paradigm, that is, an approach to design algorithms for problems that meet specific criteria (optimal substructure and overlapping sub-problems). So, it's not just ...
• 41.1k
Accepted

### Confusing point point in Dyna-Q

The hope in (f) is that the ${max}_aQ(S^\prime,a)$ has changed since it was previously evaluated so that it's effect on the present state's value can be propagated without the agent having to spend ...
• 1,447

### Confusing point in Sutton-Barto: replacing $a$ in $q(s,a)$ with a stochastic policy $\pi^\prime$

Sutton and Barto are being a bit loose with notation here. If you had a random variable $\mathbf{X}$, with individual observable values $x$ drawn from set $\mathcal{X}$ a discrete probability ...
• 32.9k
Accepted

### Unclear point in derivation of action-value function

As $q_{\pi}(s,a)$ is defined in S&B RL book as follows: we define the value of taking action a in state s under a policy $\pi$, denoted $q_{\pi}(s,a)$, as the expected return starting from $s$, ...
• 3,302
Accepted

### Sutton & Barto: Why expected square of the importance-sampling-scaled return is for policy b?

why we calculate expected return for policy $b$ instead of $\pi$? The expected return for policy $b$ is not being calculated here: \begin{align} \mathbb{E}_b \Bigg[(\prod_{t=0}^{T-1} \frac{\pi(A_t|...
• 32.9k
Accepted

### Why is there the potential problem of "learning only from the tails of episodes" in off-policy MC control?

From the same page's pseudocode for off-policy Monte Carlo Control for estimating the optimal target policy, in the nested inner loop you have: If $A_t \neq \pi(S_t)$ then exit inner Loop (proceed to ...
• 3,302
Accepted

### Why is the better policy defined with respect to all the states values being greater?

No, the answer of @foreverska is wrong, otherwise they would have said “better givena specific $\mu(s)$”. The reason is simply that given 2 policies, where one performs better than the other only in a ...
• 2,493
Accepted

### Why no falling off cliff in SARSA for the example in Sutton-Barto?

I think the point here is that Q-learning may learn the optimal policy or value function faster. The optimal policy is to choose actions that are close to the cliff, but, during learning, to behave, ...
• 41.1k

### Suppose action selection is greedy. Is Q-learning then exactly the same algorithm as Sarsa?

SARSA requires a tuple $S,A,R,S',A'$ to do an update, where $A'$ is the action you have taken at state $S'$, which means that you can only do the update once you are at state $S''$, where instead Q-...
• 2,493
Accepted

### Unclear sentence in Sutton-Barto in Temporal-Difference chapter

Underpinning an epsilon-greedy policy is a deterministic greedy policy. What they are saying here is that this underlying policy has been near optimal for quite a few episodes. They are still talking ...
• 1,447
Accepted

### How does the Belman optimality equation with altered transition probabilities in the second equation follow?

Your second equation is the BOE (Bellman optimality equation) of the altered transition probabilities for the new environment where any $\epsilon$-soft policy is moved inside as explained in the same ...
• 3,302
Accepted

### Unclear arrow in general Dyna architecture

The diagram should not be read as a formal rendering of process or architecture. It's not a UML description or similar. It is a visual aid to the text description. The large arrow shows dependency of ...
• 32.9k
Accepted

### Confusing points in Dyna-Q in Sutton-Barto about model, simulated experience and model predictions

The figure is of a broad architecture "Dyna". Of which, Dyna-Q is one such variant. So I don't think it's required that all nomenclature be exact for Dyna-Q but I will proceed to defend it....
• 1,447
Accepted

### Why is dynamic programming an example of planning?

There is no simulation in dynamic programming. In fact there is. Using the model $p(r, s'|s,a)$ (or other variations of it that are possible in Policy Iteration and Value Iteration) to predict ...
• 32.9k

### Confusing point in Sutton-Barto: replacing $a$ in $q(s,a)$ with a stochastic policy $\pi^\prime$

LHS $\pi’(s)$ is just $\pi’(.|s)$ which denotes the whole probability distribution of the said policy given state $s$, while your RHS $\pi’(a|s)$ is just a specific conditional probability when the ...
• 3,302
Accepted

### Unclear line in prioritized sweeping algorithm

$R + \gamma {max}_aQ(S^{\prime},a)$ is the target Q value for $(S,A)$. The present estimate $Q(S,A)$ is subtracted from it to arrive at the error. The absolute value is taken to arrive at the ...
• 1,447
1 vote

### How are these two terms equivalent in Sutton and Barto's derivation of the REINFORCE algorithm

In general, suppose we have a discrete random variable $X$. The expectation of $X$ is defined as $$\mathbb{E}[X] = \sum_{x\in \mathcal{X}}x \times \mathbb{P}(X = x) \; ;$$ where $\mathcal{X}$ is the ...
• 5,000
1 vote
Accepted

### How are these two terms equivalent in Sutton and Barto's derivation of the REINFORCE algorithm

Your intuition is correct as in your reference just above this equation they explain as: Notice that the right-hand side of the policy gradient theorem is a sum over states weighted by how often the ...
• 3,302
1 vote
Accepted

### Confusing convention in Sutto-Barto on Monte Carlo Tree Search: is a leaf node a state leaf node or state-action leaf node?

The answer is somewhat of a puzzle using other neighboring text from the book. We will begin by determining if the selected leaf node is a state leaf node or state-action leaf node through some of the ...
• 1,568
1 vote
Accepted

### Confusing statement in Sutton-Barto on trajectory sampling

The text is about SARSA, so yes the action values in the Q table are estimates based on-policy, on the $\epsilon$-greedy policy used for learning, with a specific value of $\epsilon$. However, this in ...
• 32.9k
1 vote

### Confusing points in Dyna-Q in Sutton-Barto about model, simulated experience and model predictions

The key property of a model is that it makes predictions of a system. Given some input - in Dyna a state and action - it provides an output, e.g. a predicted immediate reward and next state. A random ...
• 32.9k
1 vote
Accepted

### Unclear points in Dyna Maze example in Sutton-Barto

Randomness is used in epsilon greedy (both for determining when exploration should happen and what action is taken) and by "Search Control" to pick a previous state-action pair to plan for. ...
• 1,447
1 vote
Accepted

### What is the backed-up value in dynamic programming and the corresponding update based on this backed up value?

All value-based methods in Reinforcement Learning use a backup process of some kind to calculate returns or expected returns. There are multiple types of backup, but in general they consist of taking ...
• 32.9k
1 vote

### Why is the better policy defined with respect to all the states values being greater?

If the subsequent policy is better in 99 out of 100 states this would seem to be better. But imagine if the next policy in the iteration is also better in 99 out of 100 states but the one state is a ...
• 1,447
1 vote

### In Sutton-Barto a confusing point regarding $\epsilon$-soft policies in the proof for optimality of MC control without exploring starts

He's trying to turn the requirement "this policy is $\epsilon$-soft" into a feature of the environment. That way every policy in the new environment could be turned into an $\epsilon$-soft ...
• 166

Only top scored, non community-wiki answers of a minimum length are eligible