Why do we use "true labels" that are based on the output of our network in Deep Q-Learning?

Question

In the original DQN paper, the $\ell_2$ loss is taken over the distance between our network output, $\hat{q}(s_j,a_j,w)$ and the labels $y_j=r_j+\gamma \cdot \max\limits_{a'} \hat{q}(s_{j+1},a',w^-)$, where $\hat{q}(s_{j+1},a',w^-)$ is our network with static weights $w^-$, that are updated to be $w^-=w$ every $C$ steps.

This is troubling to me, as $y_j$ aren't really "true" labels as we know from supervised learning, so why should I even think that this loss updates the weights such that the output policy is something meaningful?

It seems as if my network could output some arbitrary $\hat{q}$ and with respect to this $\hat{q}$, I will try to minimize a loss, But when $\hat{q}$ isn't "optimal" per se, It is not clear that we can converge to an optimal policy.

Answer 1

The labels in DQN, and in Q-learning in general, are not "true" in the sense that they represent optimal action value functions. Instead they represent approximate action values of a current target policy.

The target policy changes every C time steps, when the network with static weights is updated. This update will include both corrections to the action value approximations, and changes to which actions are considered optimal.

The reason this converges towards an optimal action value function is related to the policy improvement theorem. With function approximation, as in DQN, the convergence is not guaranteed, but the process is stll based on the same idea. In summary it is a two-step repeated feedback process:

1. learn value function of a current policy
2. update the policy to select actions with maximum values

What this means for the TD target "labels" in DQN is:

• They are not ground truth for the optimal action value function, until after the algorithm has converged.

• They are biased, initially almost completely arbitrarily by however the target network has been initialised, and from then on due to a slowly-reducing impact from that initial bias and from lagging behind collected data.

• They are non-stationary. This means an online learning model class is required (neural networks are fine). It is also the reason why many Deep RL algorithms can suffer from catastrophic forgetting.

• When using experience replay, the TD target labels should be recalculated each time they are used.