# Why isn't it wise for us to completely erase our old Q value and replace it with the calculated Q value?

Why isn't it wise for us to completely erase our old Q value and replace it with the calculated Q value? Why can't we forget the learning rate and temporal difference?

Here's the update formula.

Underlying the temporal-difference update and many other reinforcement learning updates is the notion of policy iteration in which the estimated value function is updated to match the true value function of the current policy and the current policy is updated to be greedy with respect to the estimated value function. This process proceeds iteratively and gradually until convergence to the optimal policy and optimal value function is achieved. Gradual changes such as setting a small learning rate (e.g. $$\alpha = 0.1$$) aim to speed up convergence by lessening the frequency of the phenomenon in the above paragraph. Sutton and Barto make comments on convergence throughout their book, with the remarks surrounding line 2.7 in Section 2.5 providing a summary.