1
$\begingroup$

I was going through the Sutton book and they said the update formula for Q learning comes from the weighted average of the returns I.e

New estimate= old estimate +alpha*[returns- old estimate]

So by the law of large numbers this will converge to the optimal true q value

Now when we go to Deep Q networks,how exactly is the weighted average computed, all they simply did was try to reduce the error between the target and the estimate, and keep in mind this isn’t the true target, it’s just an unbiased estimate,since it’s an unbiased estimate how is the weighted average computed , which is the expectation?

Can someone help me out here?? Thanks in advance

$\endgroup$
1
$\begingroup$

Let's say $Q$ is the old estimate, $Q'$ the new estimate, and $R$ is the return.

We have

$$ Q' = Q + \alpha(R-Q) $$

This can be rewritten as

$$ Q' = (1-\alpha)Q + \alpha R $$

When $\alpha$ is a constant, this is an exponential weighted average of returns. If $n$ is the number of samples we get and $\alpha=1/n$ (so it decreases with each sample), we get

$$ Q' = \frac{n-1}{n}Q + \frac{1}{n}R $$

This simply represents the average return. So, playing with $\alpha$ tunes the weighting of the estimate.

| improve this answer | |
$\endgroup$
  • $\begingroup$ You’re completely missing me, I understand the weighted average in TD learning for normal Q learning, my question is how is this computed with deep Q networks ??? $\endgroup$ – Chukwudi Ogbonna Aug 22 at 13:02
  • $\begingroup$ Firstly, please don't leave me comments to check your comments. $\endgroup$ – harwiltz Aug 23 at 1:04
  • $\begingroup$ In deep Q learning, you do function approximation. You make estimates of the target returns and current Q values from samples (which is an unbiased estimator of the mean). Then you simply use SGD to minimize the MSE between your current state estimate and your target. As you mentioned, this is in fact an approximation. In general there are no convergence guarantees in deep RL. However, in the limit of infinite data, you can suppose your target network will converge to the value function (assuming the network is wide enough). $\endgroup$ – harwiltz Aug 23 at 1:10
  • $\begingroup$ I still don’t get it,I know the Q values are unbiased estimates of the optimal values, the expectation which is the mean is the weighted average, which is the update formula in normal Q learning, in Deep Q learning, how is that weighted average implemented?,that’s my question , do you understand ? $\endgroup$ – Chukwudi Ogbonna Aug 23 at 2:10
  • $\begingroup$ What I mean exactly is, in Q learning the update rule implements the Weighted average which is the expectation, in deep q learning how exactly is the weighted average implemented $\endgroup$ – Chukwudi Ogbonna Aug 23 at 2:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.