Target Network inversed in Deep Q Learning (Reinforcement Learning)

Question

I am not asking why using a Target Network is useful (this was very well explained here), but rather if using this "inversed" target network is equivalent:

$\left(r_t + \max_aQ(s_{t+1},a;\theta) - Q(s_t,a_t; \theta^-)\right)^2$ were $\theta^-$ is some old version of the parameters that gets updated every $C \in \mathbb{N}$ updates, and the Q-Network with these parameters is the target network.

Basically I'm only updating the one used to determine the expected q value at next iteration instead of only updating the one used to determine the best action. And the one I'm not updating gets updated each $C$ iteration.

Both seem to converge in my MREs but probably because the environment I'm in is "too simple" to see the benefits of a Target Network.

Answer 1

I think I'm not able to fully answer this, although I try to provide some insights:

• In regular DQN with target network, the training objective of the Q-network is to estimate the Q-value of the current state-action pair: $Q(s_t, a_t;\theta)\approx r_t + \gamma \max_{a_{t+1}}Q(s_{t+1}, a_{t+1}; \theta^-)$, where $\theta$ are the online parameters and $\theta^-$ the target parameters.
• Now if you "reverse" the target network, the quantity $\max_{a_{t+1}}Q(s_{t+1}, a_{t+1}; \theta)$ is not anymore fixed but instead it moves at each update of $\theta$ (which is usually at each timestep.) An instead you held fixed the current action-state value: $Q(s_t,a_t;\theta^-)$.

I guess this could make training more unstable because the regression target is not anymore fixed but moving: I guess this can be more problematic when $Q(s_{t+1}, a_{t+1})$ is quite different than $Q(s_t, a_t)$; although the inverse target may still be good at decorrelating.

Indeed, this should be checked on challenging environments, like Atari, in order to have a clear picture on this.