# Target Network inversed in Deep Q Learning (Reinforcement Learning)

I am not asking why using a Target Network is useful (this was very well explained here), but rather if using this "inversed" target network is equivalent:

$$\left(r_t + \max_aQ(s_{t+1},a;\theta) - Q(s_t,a_t; \theta^-)\right)^2$$ were $$\theta^-$$ is some old version of the parameters that gets updated every $$C \in \mathbb{N}$$ updates, and the Q-Network with these parameters is the target network.

Basically I'm only updating the one used to determine the expected q value at next iteration instead of only updating the one used to determine the best action. And the one I'm not updating gets updated each $$C$$ iteration.

Both seem to converge in my MREs but probably because the environment I'm in is "too simple" to see the benefits of a Target Network.

• So you're updating $\theta$ at each iteration, and $\theta^-$ every $C$ iterations, right? Apr 25 at 19:58
• Correct, this is what I'm doing. Apr 25 at 21:06

I think I'm not able to fully answer this, although I try to provide some insights:

• In regular DQN with target network, the training objective of the Q-network is to estimate the Q-value of the current state-action pair: $$Q(s_t, a_t;\theta)\approx r_t + \gamma \max_{a_{t+1}}Q(s_{t+1}, a_{t+1}; \theta^-)$$, where $$\theta$$ are the online parameters and $$\theta^-$$ the target parameters.
• Now if you "reverse" the target network, the quantity $$\max_{a_{t+1}}Q(s_{t+1}, a_{t+1}; \theta)$$ is not anymore fixed but instead it moves at each update of $$\theta$$ (which is usually at each timestep.) An instead you held fixed the current action-state value: $$Q(s_t,a_t;\theta^-)$$.

I guess this could make training more unstable because the regression target is not anymore fixed but moving: I guess this can be more problematic when $$Q(s_{t+1}, a_{t+1})$$ is quite different than $$Q(s_t, a_t)$$; although the inverse target may still be good at decorrelating.

Indeed, this should be checked on challenging environments, like Atari, in order to have a clear picture on this.