I understand we use a target network because it helps resolve issues regarding stability, however, that's not what I'm here to ask.
What I would like to understand is why a target network is used as a measure of ground truth as opposed to the expectation equation.
To clarify, here is what I mean. This is the process used for DQN:
- In DQN, we begin with a state $S$
- We then pass this state through a neural network which outputs Q values for each action in the action space
- A policy e.g. epsilon-greedy is used to take an action
- This subsequently produces the next state $S_{t+1}$
- $S_{t+1}$ is then passed through a target neural network to produce target Q values
- These target Q values are then injected into the Bellman equation which ultimately produces a target Q value via the Q-learning update rule equation
- MSE is used on 6 and 2 to compute the loss
- This is then back-propagated to update the parameters for the neural network in 2
- The target neural network has its parameters updated every X epochs to match the parameters in 2
Why do we use a target neural network to output Q values instead of using statistics. Statistics seems like a more accurate way to represent this. By statistics, I mean this:
Q values are the expected return, given the state and action under policy π.
$Q(S_{t+1},a) = V^π(S_{t+1})$ = $\mathbb{E}(r_{t+1}+ γr_{t+2}+ (γ^2)_{t+3} + ... \mid S_{t+1}) = {E}(∑γ^kr_{t+k+1}\mid S_{t+1})$
We can then take the above and inject it into the Bellman equation to update our target Q value:
$Q(S_{t},a_t) + α*(r_t+γ*max(Q(S_{t+1},a))-Q(S_{t},a))$
So, why don't we set the target to the sum of diminishing returns? Surely a target network is very inaccurate, especially since the parameters in the first few epochs for the target network are completely random.