I am working on a deep reinforcement learning problem. Throughout the episode, there is a small positive and negative reward for good or bad decisions. In the end, there is a huge reward for the completion of the episode. So, this reward function is quite sparse.
This is my understanding of how DQN works. The neural network predicts quality values for each possible action that can be taken from a state $S_1$. Let us assume the predicted quality value for an action $A$ is $Q(S_1, A)$, and this action allows the agent to reach $S_2$.
We now need the target quality value $Q_\text{target}$, so that using $Q(S_1, A)$ and $Q_\text{target}$ the temporal difference can be calculated, and updates can be made to the parameters of the value network.
$Q_\text{target}$ is composed of two terms. The immediate reward $R$ and the maximum quality value of the resulting state that this chosen action leaves us in, which can be denoted by $Q_\text{future} = \text{max}_a Q(S_2, a)$, which is in practice obtained by feeding the new state $S_2$ into the neural network and choosing (from the list of quality values for each action) the maximum quality value. We then multiply the discount factor $\gamma$ with this $Q_\text{future}$ and add it to the reward $R$, i.e. $Q_\text{target} = R + \gamma \text{max}_a Q(S_2, a) = R + \gamma Q_\text{future}$.
Now, let us assume the agent is in the penultimate state, $S_1$, and chooses the action $A$ that leads him to the completion state, $S_2$, and gets a reward $R$.
How do we form the target value $Q_\text{target}$ for $S_1$ now? Do we still include the $Q_\text{future}$ term? Or is it only the reward in this case? I am not sure if $Q_\text{future}$ even has meaning after reaching the final state $S_2$. So, I think that, for the final step, the target value must simply be the reward. Is this right?