Why isn't it wise for us to completely erase our old Q value and replace it with the calculated Q value? Why can't we forget the learning rate and temporal difference?

Here's the update formula.

The formula


1 Answer 1


Removing the learning rate will likely yield poor convergence to the optimal policy and optimal Q-values. Note that the current policy is completely dependent on the Q-values, as we take the action with highest Q-value in a given state (with a few other considerations such as exploration, etc.). If we were to remove the learning rate, then we are making a relatively large change to our Q-values and possibly to our policy as well after only a single update. For example, if the sample rewards have great variance (e.g. in stochastic environments), then drastic updates to a single Q-value may occur simply by chance when a learning rate is not used. Due to the recursive definition of Q-values, a few poor updates can undo the work of many previous updates. If this phenomenon were to occur frequently, then the policy may take a long time to converge to the optimal policy, if at all.

Underlying the temporal-difference update and many other reinforcement learning updates is the notion of policy iteration in which the estimated value function is updated to match the true value function of the current policy and the current policy is updated to be greedy with respect to the estimated value function. This process proceeds iteratively and gradually until convergence to the optimal policy and optimal value function is achieved. Gradual changes such as setting a small learning rate (e.g. $\alpha = 0.1$) aim to speed up convergence by lessening the frequency of the phenomenon in the above paragraph. Sutton and Barto make comments on convergence throughout their book, with the remarks surrounding line 2.7 in Section 2.5 providing a summary.

  • $\begingroup$ May be worth noting there are some situations in which a learning rate of 1 works quite well. That includes quite a few simple scenarios use to teach concepts, like deterministic grid worlds etc. $\endgroup$ Jun 27, 2020 at 8:08
  • $\begingroup$ @NeilSlater Since what you said is great information that would help the answer, am I supposed to edit my answer to include the info, or do I leave it as a comment? If I edit my answer to include it, do I give you attribution? I'm still a little new here and am trying to figure out the unwritten rules - I didn't see this explicitly mentioned in the rules when I just check them. $\endgroup$
    – DeepQZero
    Jun 29, 2020 at 16:34
  • $\begingroup$ The comments attached to a question or answer are generally supposed to be suggestions for improvement. They also may be deleted for any reason. So please do take my suggestion and edit something appropriate into the answer to cover it, if you agree. I don't require attribution. I also think for most cases this spoils the flow of the answer. The ideal is that the new parts are edited in so that the answer stands by itself, so no need to flag changes either. Some Stack Exchange sites do have a different culture around this, and some writers like to credit sources. Up to you here $\endgroup$ Jun 29, 2020 at 16:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .