2
$\begingroup$

In the original DQN paper, the $\ell_2$ loss is taken over the distance between our network output, $\hat{q}(s_j,a_j,w)$ and the labels $y_j=r_j+\gamma \cdot \max\limits_{a'} \hat{q}(s_{j+1},a',w^-)$, where $\hat{q}(s_{j+1},a',w^-)$ is our network with static weights $w^-$, that are updated to be $w^-=w$ every $C$ steps.

This is troubling to me, as $y_j$ aren't really "true" labels as we know from supervised learning, so why should I even think that this loss updates the weights such that the output policy is something meaningful?

It seems as if my network could output some arbitrary $\hat{q}$ and with respect to this $\hat{q}$, I will try to minimize a loss, But when $\hat{q}$ isn't "optimal" per se, It is not clear that we can converge to an optimal policy.

$\endgroup$
0

1 Answer 1

2
$\begingroup$

The labels in DQN, and in Q-learning in general, are not "true" in the sense that they represent optimal action value functions. Instead they represent approximate action values of a current target policy.

The target policy changes every C time steps, when the network with static weights is updated. This update will include both corrections to the action value approximations, and changes to which actions are considered optimal.

The reason this converges towards an optimal action value function is related to the policy improvement theorem. With function approximation, as in DQN, the convergence is not guaranteed, but the process is stll based on the same idea. In summary it is a two-step repeated feedback process:

  1. learn value function of a current policy
  2. update the policy to select actions with maximum values

What this means for the TD target "labels" in DQN is:

  • They are not ground truth for the optimal action value function, until after the algorithm has converged.

  • They are biased, initially almost completely arbitrarily by however the target network has been initialised, and from then on due to a slowly-reducing impact from that initial bias and from lagging behind collected data.

  • They are non-stationary. This means an online learning model class is required (neural networks are fine). It is also the reason why many Deep RL algorithms can suffer from catastrophic forgetting.

  • When using experience replay, the TD target labels should be recalculated each time they are used.

$\endgroup$
2
  • $\begingroup$ Thanks. Isn't the convergence property relates to the fact that we have calculated our value function based on a bellman backup process, which is a contraction map? here, from my understanding, we use a CNN which is not really related to bellman backup $\endgroup$ May 30 at 8:49
  • $\begingroup$ @HadarSharvit In a DQN, the CNN is being used within the Bellman backup process, in lieu of the Q table. $\endgroup$ May 30 at 8:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .