Share Your Experience: Take the 2024 Developer Survey

# Tag Info

## Hot answers tagged temporal-difference-methods

Accepted

### Where does the TD formula for tic-tac-toe in Sutton & Barto come from?

The equation (1) you mentioned is a simplified form of the temporal difference TD(0) update rule, specifically for the case of episodic tasks where there's no discounting $\gamma$ and the only reward ...
• 2,277

### Is the Bellman backup unbiased?

No, $B\hat{Q}$ is not an unbiased estimator. Say that $Q(s, a) = 0$, and $\hat{Q}(s, a) \sim N(0, 1)$. Then $E[max_a \space Q(s, a)] = 0$, but $E[max_a \space \hat{Q}(s, a)] > 0$ (approximately 1....
• 131
Accepted

### Is the Bellman backup unbiased?

Given your provided definition of unbiased estimator at each input data, and since $\hat{Q}$ is an unbiased estimator for the true action value $Q$ at each state-action input pair, applying Bellman ...
• 2,277
Accepted

### Unclear sentence in Sutton-Barto in Temporal-Difference chapter

Underpinning an epsilon-greedy policy is a deterministic greedy policy. What they are saying here is that this underlying policy has been near optimal for quite a few episodes. They are still talking ...
• 1,263
1 vote

### Convergence of epsilon greedy policy (with no epsilon decay) using TD Learning?

In general on-policy Sarsa algo with your epsilon greedy behavior policy actually is the action value version of TD(0) algo, thus it has the same convergence property of TD(0) and is confirmed in the ...
• 2,277
1 vote

### Why does my implementation of TD(0) not work?

It is a very simple example with terminal states s2 and s3 and it is clear that the Value function should converge to: ...
• 32.4k
1 vote

### How to calculate TD error?

In the tabular case (i.e. discrete states), you just look-up $V(s_{t+1})$ (note that $t$ is the subscript of the state $s$) in the $V$ table, like V[s[t + 1]]. ...
• 2,948
1 vote
Accepted

### TD Leaf value function update

In temporal difference learning, we do not wait to collect the full trajectory before performing backups. That means at the time of performing the backup, we may be "wrong". In fact that is ...
• 32.4k

Only top scored, non community-wiki answers of a minimum length are eligible