Skip to main content
Share Your Experience: Take the 2024 Developer Survey
4 votes
Accepted

Where does the TD formula for tic-tac-toe in Sutton & Barto come from?

The equation (1) you mentioned is a simplified form of the temporal difference TD(0) update rule, specifically for the case of episodic tasks where there's no discounting $\gamma$ and the only reward ...
cinch's user avatar
  • 2,277
3 votes

Is the Bellman backup unbiased?

No, $B\hat{Q}$ is not an unbiased estimator. Say that $Q(s, a) = 0$, and $\hat{Q}(s, a) \sim N(0, 1)$. Then $E[max_a \space Q(s, a)] = 0$, but $E[max_a \space \hat{Q}(s, a)] > 0$ (approximately 1....
user1502040's user avatar
3 votes
Accepted

Is the Bellman backup unbiased?

Given your provided definition of unbiased estimator at each input data, and since $\hat{Q}$ is an unbiased estimator for the true action value $Q$ at each state-action input pair, applying Bellman ...
cinch's user avatar
  • 2,277
2 votes
Accepted

Unclear sentence in Sutton-Barto in Temporal-Difference chapter

Underpinning an epsilon-greedy policy is a deterministic greedy policy. What they are saying here is that this underlying policy has been near optimal for quite a few episodes. They are still talking ...
foreverska's user avatar
  • 1,263
1 vote

Convergence of epsilon greedy policy (with no epsilon decay) using TD Learning?

In general on-policy Sarsa algo with your epsilon greedy behavior policy actually is the action value version of TD(0) algo, thus it has the same convergence property of TD(0) and is confirmed in the ...
cinch's user avatar
  • 2,277
1 vote

Why does my implementation of TD(0) not work?

It is a very simple example with terminal states s2 and s3 and it is clear that the Value function should converge to: ...
Neil Slater's user avatar
  • 32.4k
1 vote

How to calculate TD error?

In the tabular case (i.e. discrete states), you just look-up $V(s_{t+1})$ (note that $t$ is the subscript of the state $s$) in the $V$ table, like V[s[t + 1]]. ...
Luca Anzalone's user avatar
1 vote
Accepted

TD Leaf value function update

In temporal difference learning, we do not wait to collect the full trajectory before performing backups. That means at the time of performing the backup, we may be "wrong". In fact that is ...
Neil Slater's user avatar
  • 32.4k

Only top scored, non community-wiki answers of a minimum length are eligible