New answers tagged

4 votes
Accepted

Where does the TD formula for tic-tac-toe in Sutton & Barto come from?

The equation (1) you mentioned is a simplified form of the temporal difference TD(0) update rule, specifically for the case of episodic tasks where there's no discounting $\gamma$ and the only reward ...
cinch's user avatar
  • 2,307
3 votes

Is the Bellman backup unbiased?

No, $B\hat{Q}$ is not an unbiased estimator. Say that $Q(s, a) = 0$, and $\hat{Q}(s, a) \sim N(0, 1)$. Then $E[max_a \space Q(s, a)] = 0$, but $E[max_a \space \hat{Q}(s, a)] > 0$ (approximately 1....
user1502040's user avatar
3 votes
Accepted

Is the Bellman backup unbiased?

Given your provided definition of unbiased estimator at each input data, and since $\hat{Q}$ is an unbiased estimator for the true action value $Q$ at each state-action input pair, applying Bellman ...
cinch's user avatar
  • 2,307

Top 50 recent answers are included