Share Your Experience: Take the 2024 Developer Survey

# Tag Info

1 vote

### tic-tac-toe - tabular q-learning - what is the formula to calculate the number of entries in the q-table

Great answer from @Neil Slater; however, I wanted to give my 2 cents. Even though it's called "tabular" q-learning, it doesn't imply that you have to use a table to access it; yes, a table ...
• 2,173
1 vote
Accepted

### tic-tac-toe - tabular q-learning - what is the formula to calculate the number of entries in the q-table

The bad news is that there is no simple formula for calculating the state space or state+action space here. It is relatively easy to get upper or lower bounds to it, but complex to precisely calculate....
• 32.5k
1 vote
Accepted

### Confusing statement in Sutton-Barto on trajectory sampling

The text is about SARSA, so yes the action values in the Q table are estimates based on-policy, on the $\epsilon$-greedy policy used for learning, with a specific value of $\epsilon$. However, this in ...
• 32.5k

### Violation of Markov property

Markovian environments contain everything the agent needs to make an optimal decision in the present state. There cannot be a state where an agent passes through it once and the optimal action is one ...
• 1,263
Accepted

### Can we implement a memory in a REINFORCE algorithm for RL?

Using a "memory" of previous experience with the REINFORCE algorithm will not work. The algorithm relies heavily on the training data distribution matching current on-policy behaviour. ...
• 32.5k
1 vote

### Why must the value of a state under an optimal policy equal the expected return for the best action from that state?

I was also confused when Sutton and Barto claimed in chapter 3, as you stated, that the value of a state under an optimal policy must equal the expected return for the best action from that state. ...
• 111
Accepted

### Why is dynamic programming an example of planning?

There is no simulation in dynamic programming. In fact there is. Using the model $p(r, s'|s,a)$ (or other variations of it that are possible in Policy Iteration and Value Iteration) to predict ...
• 32.5k

### Why is dynamic programming an example of planning?

Dynamic programming is a algorithm paradigm, that is, an approach to design algorithms for problems that meet specific criteria (optimal substructure and overlapping sub-problems). So, it's not just ...
• 40.8k

### Why is importance sampling ratio in n-step TD multiplying error rather than return n-step return?

No, also for MC Control the importance sampling weights the update, and not just the return But the reason I think can be understood intuitively as: if you have the same policy as the behavioral, ...
• 2,173
Accepted

### Unclear arrow in general Dyna architecture

The diagram should not be read as a formal rendering of process or architecture. It's not a UML description or similar. It is a visual aid to the text description. The large arrow shows dependency of ...
• 32.5k
1 vote

### Confusing points in Dyna-Q in Sutton-Barto about model, simulated experience and model predictions

The key property of a model is that it makes predictions of a system. Given some input - in Dyna a state and action - it provides an output, e.g. a predicted immediate reward and next state. A random ...
• 32.5k
Accepted

### Confusing point point in Dyna-Q

The hope in (f) is that the ${max}_aQ(S^\prime,a)$ has changed since it was previously evaluated so that it's effect on the present state's value can be propagated without the agent having to spend ...
• 1,263
Accepted

### Confusing points in Dyna-Q in Sutton-Barto about model, simulated experience and model predictions

The figure is of a broad architecture "Dyna". Of which, Dyna-Q is one such variant. So I don't think it's required that all nomenclature be exact for Dyna-Q but I will proceed to defend it....
• 1,263
1 vote
Accepted

### Unclear points in Dyna Maze example in Sutton-Barto

Randomness is used in epsilon greedy (both for determining when exploration should happen and what action is taken) and by "Search Control" to pick a previous state-action pair to plan for. ...
• 1,263
1 vote
Accepted

### What is the backed-up value in dynamic programming and the corresponding update based on this backed up value?

All value-based methods in Reinforcement Learning use a backup process of some kind to calculate returns or expected returns. There are multiple types of backup, but in general they consist of taking ...
• 32.5k

### Neural network for specific numbers from a range (Q learning)

def action(self, state): if np.random.rand() > self.epsilon: return np.random.randint(0,4) return np.argmax(self.model.predict(state)) You do ...
• 1,263
Accepted

### Neural network for specific numbers from a range (Q learning)

I am not able to understand how can I tell the neural network that it can take only one of the following values You don't have to. The neural network in Deep Q Learning (the DQN) is not configured to ...
• 32.5k
Accepted

### Why no falling off cliff in SARSA for the example in Sutton-Barto?

I think the point here is that Q-learning may learn the optimal policy or value function faster. The optimal policy is to choose actions that are close to the cliff, but, during learning, to behave, ...
• 40.8k

### Why is there no exploration-optimality trade-off in Q-learning?

Whilst it's not perfect separation, because you may still pay real costs for exploration during learning (in fact possibly worse than on-policy methods), then off-policy methods do offer a clean way ...
• 32.5k

### Why is there no exploration-optimality trade-off in Q-learning?

I feel that this question is more easily understood by contrasting on-policy methods with off-policy methods. An off-policy method generates data in the environment with a behavior policy during ...
• 1,399

### $\gamma^t$ in REINFORCE update (Sutton-Barto RL book Exercise 13.2)

We just need to exchange the order of summation and then sample episodes in both space and time to perform gradient descent. My response is a bit late, but I hope it can still be of help to you.
• 1

### Value iteration in a Grid World Example

The expected immediate reward functions $R(s)$ and $R(s,a)$ are convenient structures for the more fundamental $R(s,a,s')$ (expected reward for starting in state $s$, taking action $a$ and arriving in ...
• 32.5k
SARSA requires a tuple $S,A,R,S',A'$ to do an update, where $A'$ is the action you have taken at state $S'$, which means that you can only do the update once you are at state $S''$, where instead Q-...