I am working on a deep reinforcement learning problem, when I got stuck at the following questions. They are rather general and not specific to my specific problem. The solution uses a sparse reward structure. Throughout the episode there is a small positive and negative reward for good or bad decisions. At the end there is a huge reward for completion of the episode.
This is my understanding of how DQN works. The neural network predicts quality values for each possible action that can be taken from a state (S1). Let us assume the predicted quality value for an action A is Q1 and this action allows the agent to reach S2.
We now need the target quality value Qt, so that using Q1 and Qt gradients can be calculated and updates can be made to the parameters of the network. Qt is calculated using the Bellman equation. For an arbitrary state the bellman equation consists of two parts. The immediate reward:R and the maximum quality value of the resulting state that this chosen action leaves us in (say Q_future).
Q_future is obtained by feeding the new state S2 into the neural network and choosing (from the list of quality values for each action) the maximum quality value.
We then apply a discount to this Q_future and add it to the reward R.
Now let us assume the agent is in the penultimate state (S1) and chooses the action (A) that leads him to completion state (S2) and gets him the huge final reward :Rmax. My question is how do we form the target value for this case? Do we still include the Q_future term? Or is it only the reward in this case?
The agent would not have seen states beyond the final state. So I am not sure if Q_future even has a meaning after reaching the final state. But if we still consider this Q_future mathematically, to be the output of the network when the final state is plugged in, then I would expect this to be comparable or greater than the predicted Q. So the target Q becomes a huge reward + (discount_factor*Q_future). This does not seem to converge as whatever this predicted Q is in the final step, the target Q is much bigger. So I feel for the final step, the target value must simply be the reward. Is this right?