The labels in DQN, and in Q-learning in general, are *not* "true" in the sense that they represent optimal action value functions. Instead they represent approximate action values of a *current* target policy. The target policy changes every C time steps, when the network with static weights is updated. This update will include both corrections to the action value approximations, and changes to which actions are considered optimal. The reason this converges towards an optimal action value function is related to the [policy improvement theorem][1]. With function approximation, as in DQN, the convergence is not guaranteed, but the process is stll based on the same idea. In summary it is a two-step repeated feedback process: 1. learn value function of a current policy 1. update the policy to select actions with maximum values What this means for the TD target "labels" in DQN is: * They are not ground truth for the optimal action value function, until *after* the algorithm has converged. * They are biased, initially almost completely arbitrarily by however the target network has been initialised, and from then on due to a slowly-reducing impact from that initial bias and from lagging behind collected data. * They are non-stationary. This means an online learning model class is required (neural networks are fine). It is also the reason why many Deep RL algorithms can suffer from catastrophic forgetting. * When using experience replay, the TD target labels should be recalculated each time they are used. [1]: http://incompleteideas.net/book/ebook/node42.html