I'm trying to understand DQN. I understand where the loss function comes from. I'm just unsure about why the target function works in practice. Given the loss function $$ L_i(\theta_i) = [(y_i - Q(s,a;\theta_i))^2] $$ where $$ y_i = r + \gamma * max_{a'}Q(s',a';\theta_{i-1}) $$ is the target value in the loss function.

From my understanding, experiences are pulled from the replay buffer, then the DQN is used to estimate the future sum of the discounted rewards for the next state (assuming it plays optimally) and adds this onto the current rewards $r$ to create the target value. Then the DQN is used again to estimate $Q$ value for the current state. Then the loss function is just the difference between the target and the estimated $Q$ value for the current state. Afterward, you optimize the loss function.

But, if the parameters of the DQN start off randomly, then surely the target value will be completely wrong since the parameters that define that target function are random. So, if the target function is wrong, then it will minimize the difference between the target value and the estimated value, but it will be learning to predict incorrect values, since the target values are wrong?

I don't understand why the target value works if the parameters of the DQN needed to create that target are completely random.

What obvious mistake am I making here?

  • $\begingroup$ Normally when calculating the target $y_i$, we should use parameters/weights $\theta^{-}$ of the target network, which are a periodically-updated copy of the main network parameters $\theta$ (See here). So, just checking if this is a typo in your notation, and if even when fixed, your doubts still exist. $\endgroup$ – user5093249 Jun 2 '20 at 18:44
  • $\begingroup$ Note that, in the case of tabular Q-learning (Q-learning without function approximation), the initial targets are also noisy, but hopefully they will converge to the correct target, as you update the state-action value function and explore (and exploit) the environment. $\endgroup$ – nbro Jun 4 '20 at 14:43
  • $\begingroup$ Thank you for the replies. I think I have a better idea about what is happening. $\endgroup$ – Alex Obrien Jun 4 '20 at 16:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.