4
votes
Accepted
Where does the TD formula for tic-tac-toe in Sutton & Barto come from?
The equation (1) you mentioned is a simplified form of the temporal difference TD(0) update rule, specifically for the case of episodic tasks where there's no discounting $\gamma$ and the only reward ...
- 3,302
4
votes
Accepted
Unclear paragraph in Sutton-Barto on exploration/exploitation relating to bandit like decision tasks
The quoted text is from the end of a paragraph that is explaining some aspects of bandit algorithms, and looking forward to how knowledge of them is applicable in a reinforcement learning context. ...
- 32.9k
3
votes
Why is dynamic programming an example of planning?
Dynamic programming is a algorithm paradigm, that is, an approach to design algorithms for problems that meet specific criteria (optimal substructure and overlapping sub-problems). So, it's not just ...
- 41.1k
3
votes
Accepted
Confusing point point in Dyna-Q
The hope in (f) is that the ${max}_aQ(S^\prime,a)$ has changed since it was previously evaluated so that it's effect on the present state's value can be propagated without the agent having to spend ...
- 1,447
3
votes
Confusing point in Sutton-Barto: replacing $a$ in $q(s,a)$ with a stochastic policy $\pi^\prime$
Sutton and Barto are being a bit loose with notation here.
If you had
a random variable $\mathbf{X}$, with individual observable values $x$ drawn from set $\mathcal{X}$
a discrete probability ...
- 32.9k
2
votes
Accepted
Unclear point in derivation of action-value function
As $q_{\pi}(s,a)$ is defined in S&B RL book as follows:
we define the value of taking action a in state s under a policy $\pi$, denoted $q_{\pi}(s,a)$, as the expected return starting from $s$, ...
- 3,302
2
votes
Accepted
Sutton & Barto: Why expected square of the importance-sampling-scaled return is for policy b?
why we calculate expected return for policy $b$ instead of $\pi$?
The expected return for policy $b$ is not being calculated here:
\begin{align}
\mathbb{E}_b \Bigg[(\prod_{t=0}^{T-1} \frac{\pi(A_t|...
- 32.9k
2
votes
Accepted
Why is there the potential problem of "learning only from the tails of episodes" in off-policy MC control?
From the same page's pseudocode for off-policy Monte Carlo Control for estimating the optimal target policy, in the nested inner loop you have:
If $A_t \neq \pi(S_t)$ then exit inner Loop (proceed to ...
- 3,302
2
votes
Accepted
Why is the better policy defined with respect to all the states values being greater?
No, the answer of @foreverska is wrong, otherwise they would have said “better givena specific $\mu(s)$”. The reason is simply that given 2 policies, where one performs better than the other only in a ...
- 2,493
2
votes
Accepted
Why no falling off cliff in SARSA for the example in Sutton-Barto?
I think the point here is that Q-learning may learn the optimal policy or value function faster. The optimal policy is to choose actions that are close to the cliff, but, during learning, to behave, ...
- 41.1k
2
votes
Suppose action selection is greedy. Is Q-learning then exactly the same algorithm as Sarsa?
SARSA requires a tuple $S,A,R,S',A'$ to do an update, where $A'$ is the action you have taken at state $S'$, which means that you can only do the update once you are at state $S''$, where instead Q-...
- 2,493
2
votes
Accepted
Unclear sentence in Sutton-Barto in Temporal-Difference chapter
Underpinning an epsilon-greedy policy is a deterministic greedy policy. What they are saying here is that this underlying policy has been near optimal for quite a few episodes.
They are still talking ...
- 1,447
2
votes
Accepted
How does the Belman optimality equation with altered transition probabilities in the second equation follow?
Your second equation is the BOE (Bellman optimality equation) of the altered transition probabilities for the new environment where any $\epsilon$-soft policy is moved inside as explained in the same ...
- 3,302
2
votes
Accepted
Unclear arrow in general Dyna architecture
The diagram should not be read as a formal rendering of process or architecture. It's not a UML description or similar. It is a visual aid to the text description.
The large arrow shows dependency of ...
- 32.9k
2
votes
Accepted
Confusing points in Dyna-Q in Sutton-Barto about model, simulated experience and model predictions
The figure is of a broad architecture "Dyna". Of which, Dyna-Q is one such variant. So I don't think it's required that all nomenclature be exact for Dyna-Q but I will proceed to defend it....
- 1,447
2
votes
Accepted
Why is dynamic programming an example of planning?
There is no simulation in dynamic programming.
In fact there is.
Using the model $p(r, s'|s,a)$ (or other variations of it that are possible in Policy Iteration and Value Iteration) to predict ...
- 32.9k
2
votes
Confusing point in Sutton-Barto: replacing $a$ in $q(s,a)$ with a stochastic policy $\pi^\prime$
LHS $\pi’(s)$ is just $\pi’(.|s)$ which denotes the whole probability distribution of the said policy given state $s$, while your RHS $\pi’(a|s)$ is just a specific conditional probability when the ...
- 3,302
2
votes
Accepted
Unclear line in prioritized sweeping algorithm
$ R + \gamma {max}_aQ(S^{\prime},a) $ is the target Q value for $(S,A)$. The present estimate $Q(S,A)$ is subtracted from it to arrive at the error. The absolute value is taken to arrive at the ...
- 1,447
1
vote
How are these two terms equivalent in Sutton and Barto's derivation of the REINFORCE algorithm
In general, suppose we have a discrete random variable $X$. The expectation of $X$ is defined as
$$\mathbb{E}[X] = \sum_{x\in \mathcal{X}}x \times \mathbb{P}(X = x) \; ;$$
where $\mathcal{X}$ is the ...
- 5,000
1
vote
Accepted
How are these two terms equivalent in Sutton and Barto's derivation of the REINFORCE algorithm
Your intuition is correct as in your reference just above this equation they explain as:
Notice that the right-hand side of the policy gradient theorem is a sum over states weighted by how often the ...
- 3,302
1
vote
Accepted
Confusing convention in Sutto-Barto on Monte Carlo Tree Search: is a leaf node a state leaf node or state-action leaf node?
The answer is somewhat of a puzzle using other neighboring text from the book. We will begin by determining if the selected leaf node is a state leaf node or state-action leaf node through some of the ...
- 1,568
1
vote
Accepted
Confusing statement in Sutton-Barto on trajectory sampling
The text is about SARSA, so yes the action values in the Q table are estimates based on-policy, on the $\epsilon$-greedy policy used for learning, with a specific value of $\epsilon$.
However, this in ...
- 32.9k
1
vote
Confusing points in Dyna-Q in Sutton-Barto about model, simulated experience and model predictions
The key property of a model is that it makes predictions of a system. Given some input - in Dyna a state and action - it provides an output, e.g. a predicted immediate reward and next state.
A random ...
- 32.9k
1
vote
Accepted
Unclear points in Dyna Maze example in Sutton-Barto
Randomness is used in epsilon greedy (both for determining when exploration should happen and what action is taken) and by "Search Control" to pick a previous state-action pair to plan for. ...
- 1,447
1
vote
Accepted
What is the backed-up value in dynamic programming and the corresponding update based on this backed up value?
All value-based methods in Reinforcement Learning use a backup process of some kind to calculate returns or expected returns.
There are multiple types of backup, but in general they consist of taking ...
- 32.9k
1
vote
Why is the better policy defined with respect to all the states values being greater?
If the subsequent policy is better in 99 out of 100 states this would seem to be better. But imagine if the next policy in the iteration is also better in 99 out of 100 states but the one state is a ...
- 1,447
1
vote
In Sutton-Barto a confusing point regarding $\epsilon$-soft policies in the proof for optimality of MC control without exploring starts
He's trying to turn the requirement "this policy is $\epsilon$-soft" into a feature of the environment. That way every policy in the new environment could be turned into an $\epsilon$-soft ...
- 166
Only top scored, non community-wiki answers of a minimum length are eligible