Skip to main content
Share Your Experience: Take the 2024 Developer Survey

New answers tagged

1 vote
Accepted

Confusing statement in Sutton-Barto on trajectory sampling

The text is about SARSA, so yes the action values in the Q table are estimates based on-policy, on the $\epsilon$-greedy policy used for learning, with a specific value of $\epsilon$. However, this in ...
Neil Slater's user avatar
  • 32.4k
2 votes
Accepted

Why is dynamic programming an example of planning?

There is no simulation in dynamic programming. In fact there is. Using the model $p(r, s'|s,a)$ (or other variations of it that are possible in Policy Iteration and Value Iteration) to predict ...
Neil Slater's user avatar
  • 32.4k
3 votes

Why is dynamic programming an example of planning?

Dynamic programming is a algorithm paradigm, that is, an approach to design algorithms for problems that meet specific criteria (optimal substructure and overlapping sub-problems). So, it's not just ...
nbro's user avatar
  • 40.8k
2 votes
Accepted

Unclear arrow in general Dyna architecture

The diagram should not be read as a formal rendering of process or architecture. It's not a UML description or similar. It is a visual aid to the text description. The large arrow shows dependency of ...
Neil Slater's user avatar
  • 32.4k
1 vote

Confusing points in Dyna-Q in Sutton-Barto about model, simulated experience and model predictions

The key property of a model is that it makes predictions of a system. Given some input - in Dyna a state and action - it provides an output, e.g. a predicted immediate reward and next state. A random ...
Neil Slater's user avatar
  • 32.4k
3 votes
Accepted

Confusing point point in Dyna-Q

The hope in (f) is that the ${max}_aQ(S^\prime,a)$ has changed since it was previously evaluated so that it's effect on the present state's value can be propagated without the agent having to spend ...
foreverska's user avatar
  • 1,263
2 votes
Accepted

Confusing points in Dyna-Q in Sutton-Barto about model, simulated experience and model predictions

The figure is of a broad architecture "Dyna". Of which, Dyna-Q is one such variant. So I don't think it's required that all nomenclature be exact for Dyna-Q but I will proceed to defend it....
foreverska's user avatar
  • 1,263
1 vote
Accepted

Unclear points in Dyna Maze example in Sutton-Barto

Randomness is used in epsilon greedy (both for determining when exploration should happen and what action is taken) and by "Search Control" to pick a previous state-action pair to plan for. ...
foreverska's user avatar
  • 1,263
1 vote
Accepted

What is the backed-up value in dynamic programming and the corresponding update based on this backed up value?

All value-based methods in Reinforcement Learning use a backup process of some kind to calculate returns or expected returns. There are multiple types of backup, but in general they consist of taking ...
Neil Slater's user avatar
  • 32.4k
2 votes
Accepted

Why no falling off cliff in SARSA for the example in Sutton-Barto?

I think the point here is that Q-learning may learn the optimal policy or value function faster. The optimal policy is to choose actions that are close to the cliff, but, during learning, to behave, ...
nbro's user avatar
  • 40.8k
0 votes

$\gamma^t$ in REINFORCE update (Sutton-Barto RL book Exercise 13.2)

We just need to exchange the order of summation and then sample episodes in both space and time to perform gradient descent. My response is a bit late, but I hope it can still be of help to you.
Jax's user avatar
  • 1
2 votes

Suppose action selection is greedy. Is Q-learning then exactly the same algorithm as Sarsa?

SARSA requires a tuple $S,A,R,S',A'$ to do an update, where $A'$ is the action you have taken at state $S'$, which means that you can only do the update once you are at state $S''$, where instead Q-...
Alberto's user avatar
  • 2,153
2 votes
Accepted

Unclear sentence in Sutton-Barto in Temporal-Difference chapter

Underpinning an epsilon-greedy policy is a deterministic greedy policy. What they are saying here is that this underlying policy has been near optimal for quite a few episodes. They are still talking ...
foreverska's user avatar
  • 1,263

Top 50 recent answers are included