Skip to main content
Share Your Experience: Take the 2024 Developer Survey

New answers tagged

1 vote

tic-tac-toe - tabular q-learning - what is the formula to calculate the number of entries in the q-table

Great answer from @Neil Slater; however, I wanted to give my 2 cents. Even though it's called "tabular" q-learning, it doesn't imply that you have to use a table to access it; yes, a table ...
Alberto's user avatar
  • 2,173
1 vote
Accepted

tic-tac-toe - tabular q-learning - what is the formula to calculate the number of entries in the q-table

The bad news is that there is no simple formula for calculating the state space or state+action space here. It is relatively easy to get upper or lower bounds to it, but complex to precisely calculate....
Neil Slater's user avatar
  • 32.5k
1 vote
Accepted

Confusing statement in Sutton-Barto on trajectory sampling

The text is about SARSA, so yes the action values in the Q table are estimates based on-policy, on the $\epsilon$-greedy policy used for learning, with a specific value of $\epsilon$. However, this in ...
Neil Slater's user avatar
  • 32.5k
0 votes

Violation of Markov property

Markovian environments contain everything the agent needs to make an optimal decision in the present state. There cannot be a state where an agent passes through it once and the optimal action is one ...
foreverska's user avatar
  • 1,263
5 votes
Accepted

Can we implement a memory in a REINFORCE algorithm for RL?

Using a "memory" of previous experience with the REINFORCE algorithm will not work. The algorithm relies heavily on the training data distribution matching current on-policy behaviour. ...
Neil Slater's user avatar
  • 32.5k
1 vote

Why must the value of a state under an optimal policy equal the expected return for the best action from that state?

I was also confused when Sutton and Barto claimed in chapter 3, as you stated, that the value of a state under an optimal policy must equal the expected return for the best action from that state. ...
AMT's user avatar
  • 111
2 votes
Accepted

Why is dynamic programming an example of planning?

There is no simulation in dynamic programming. In fact there is. Using the model $p(r, s'|s,a)$ (or other variations of it that are possible in Policy Iteration and Value Iteration) to predict ...
Neil Slater's user avatar
  • 32.5k
3 votes

Why is dynamic programming an example of planning?

Dynamic programming is a algorithm paradigm, that is, an approach to design algorithms for problems that meet specific criteria (optimal substructure and overlapping sub-problems). So, it's not just ...
nbro's user avatar
  • 40.8k
0 votes

Why is importance sampling ratio in n-step TD multiplying error rather than return n-step return?

No, also for MC Control the importance sampling weights the update, and not just the return But the reason I think can be understood intuitively as: if you have the same policy as the behavioral, ...
Alberto's user avatar
  • 2,173
2 votes
Accepted

Unclear arrow in general Dyna architecture

The diagram should not be read as a formal rendering of process or architecture. It's not a UML description or similar. It is a visual aid to the text description. The large arrow shows dependency of ...
Neil Slater's user avatar
  • 32.5k
1 vote

Confusing points in Dyna-Q in Sutton-Barto about model, simulated experience and model predictions

The key property of a model is that it makes predictions of a system. Given some input - in Dyna a state and action - it provides an output, e.g. a predicted immediate reward and next state. A random ...
Neil Slater's user avatar
  • 32.5k
3 votes
Accepted

Confusing point point in Dyna-Q

The hope in (f) is that the ${max}_aQ(S^\prime,a)$ has changed since it was previously evaluated so that it's effect on the present state's value can be propagated without the agent having to spend ...
foreverska's user avatar
  • 1,263
2 votes
Accepted

Confusing points in Dyna-Q in Sutton-Barto about model, simulated experience and model predictions

The figure is of a broad architecture "Dyna". Of which, Dyna-Q is one such variant. So I don't think it's required that all nomenclature be exact for Dyna-Q but I will proceed to defend it....
foreverska's user avatar
  • 1,263
1 vote
Accepted

Unclear points in Dyna Maze example in Sutton-Barto

Randomness is used in epsilon greedy (both for determining when exploration should happen and what action is taken) and by "Search Control" to pick a previous state-action pair to plan for. ...
foreverska's user avatar
  • 1,263
1 vote
Accepted

What is the backed-up value in dynamic programming and the corresponding update based on this backed up value?

All value-based methods in Reinforcement Learning use a backup process of some kind to calculate returns or expected returns. There are multiple types of backup, but in general they consist of taking ...
Neil Slater's user avatar
  • 32.5k
2 votes

Neural network for specific numbers from a range (Q learning)

def action(self, state): if np.random.rand() > self.epsilon: return np.random.randint(0,4) return np.argmax(self.model.predict(state)) You do ...
foreverska's user avatar
  • 1,263
2 votes
Accepted

Neural network for specific numbers from a range (Q learning)

I am not able to understand how can I tell the neural network that it can take only one of the following values You don't have to. The neural network in Deep Q Learning (the DQN) is not configured to ...
Neil Slater's user avatar
  • 32.5k
2 votes
Accepted

Why no falling off cliff in SARSA for the example in Sutton-Barto?

I think the point here is that Q-learning may learn the optimal policy or value function faster. The optimal policy is to choose actions that are close to the cliff, but, during learning, to behave, ...
nbro's user avatar
  • 40.8k
2 votes

Why is there no exploration-optimality trade-off in Q-learning?

Whilst it's not perfect separation, because you may still pay real costs for exploration during learning (in fact possibly worse than on-policy methods), then off-policy methods do offer a clean way ...
Neil Slater's user avatar
  • 32.5k
2 votes

Why is there no exploration-optimality trade-off in Q-learning?

I feel that this question is more easily understood by contrasting on-policy methods with off-policy methods. An off-policy method generates data in the environment with a behavior policy during ...
DeepQZero's user avatar
  • 1,399
0 votes

$\gamma^t$ in REINFORCE update (Sutton-Barto RL book Exercise 13.2)

We just need to exchange the order of summation and then sample episodes in both space and time to perform gradient descent. My response is a bit late, but I hope it can still be of help to you.
Jax's user avatar
  • 1
0 votes

Value iteration in a Grid World Example

The expected immediate reward functions $R(s)$ and $R(s,a)$ are convenient structures for the more fundamental $R(s,a,s')$ (expected reward for starting in state $s$, taking action $a$ and arriving in ...
Neil Slater's user avatar
  • 32.5k
2 votes

It is not clear why SARSA is on-policy but Q-learning off-policy

Your mistake is your definition of on-policy. In both cases, we select actions based on our current action value estimates. It's how we select those actions that differentiates the 2 algorithms. An ...
nbro's user avatar
  • 40.8k
2 votes

Suppose action selection is greedy. Is Q-learning then exactly the same algorithm as Sarsa?

SARSA requires a tuple $S,A,R,S',A'$ to do an update, where $A'$ is the action you have taken at state $S'$, which means that you can only do the update once you are at state $S''$, where instead Q-...
Alberto's user avatar
  • 2,173

Top 50 recent answers are included