Target network in DQN is known to make the network more stable, and the loss is like "how good I'm now compared to using the target". What I don't understand is, if the target network is the stable one, why do we keep using/saving the first model as the predictor instead of the target?
I see in the code everywhere:
At the end, the model is saved and used for prediction and not the target.
The learning model and target model are only different by N steps (typically a few thousand) out the entire taining process. If the process is near complete, they will also be quite similar.
The target model is not inherently more stable in terms of producing "correct" or "better" Q values. Instead it is kept static for a period of time in order to stabilise the temporal difference (TD) target $R_{t+1} + \gamma \text{max}_{a'} \hat{q}(S_{t+1}, a', \theta^{-})$
Due to the copying stage that you listed:
If you return the target model, this is identical to returning the learning model from N steps beforehand.
Which in turn means that there was no point in doing the last N steps of training. You may as well have returned the learning model at the same point as copying.
Or another way of thinking about it: Returning the learning model at the end of the training process is identical to copying it to the target model as normal, then returning the target model immediately.
Target network is not more stable. Both networks are the same in the regard that no one is more stable than the other. The reason for using a target network is that your current network after each step is updated. So, by not using a target network and using just the current network, after each update the rewards for many states will be modified slightly. So, for a particular state, after each update, the reward will be modified which will lead to an unstable reward. This happens because when you update Q value function for state S, you may also slightly change the reward it predicts for state S' in the next step. So, each update to Q value function changes slightly the rewards for many other states, which makes the predicted rewards in all those states to be slightly unstable, as they are changed very often and you dont have a clear reward in those states.
However, by using a target network, your reward will be more stable(as you dont update your target network at each step, but you only update the current network).
So, by using a target network, the update is more stable, not the individual networks. And because the target network is an older version of the current network, it is sensible to use the current network as it was trained more than the target network
