I was going through the Sutton book and they said the update formula for Q learning comes from the weighted average of the returns I.e
New estimate= old estimate +alpha*[returns- old estimate]
So by the law of large numbers this will converge to the optimal true q value
Now when we go to Deep Q networks,how exactly is the weighted average computed, all they simply did was try to reduce the error between the target and the estimate, and keep in mind this isn’t the true target, it’s just an unbiased estimate,since it’s an unbiased estimate how is the weighted average computed , which is the expectation?
Can someone help me out here?? Thanks in advance