# How do we compute the target value when the agent ends up in the terminal state?

I am working on a deep reinforcement learning problem. Throughout the episode, there is a small positive and negative reward for good or bad decisions. In the end, there is a huge reward for the completion of the episode. So, this reward function is quite sparse.

This is my understanding of how DQN works. The neural network predicts quality values for each possible action that can be taken from a state $$S_1$$. Let us assume the predicted quality value for an action $$A$$ is $$Q(S_1, A)$$, and this action allows the agent to reach $$S_2$$.

We now need the target quality value $$Q_\text{target}$$, so that using $$Q(S_1, A)$$ and $$Q_\text{target}$$ the temporal difference can be calculated, and updates can be made to the parameters of the value network.

$$Q_\text{target}$$ is composed of two terms. The immediate reward $$R$$ and the maximum quality value of the resulting state that this chosen action leaves us in, which can be denoted by $$Q_\text{future} = \text{max}_a Q(S_2, a)$$, which is in practice obtained by feeding the new state $$S_2$$ into the neural network and choosing (from the list of quality values for each action) the maximum quality value. We then multiply the discount factor $$\gamma$$ with this $$Q_\text{future}$$ and add it to the reward $$R$$, i.e. $$Q_\text{target} = R + \gamma \text{max}_a Q(S_2, a) = R + \gamma Q_\text{future}$$.

Now, let us assume the agent is in the penultimate state, $$S_1$$, and chooses the action $$A$$ that leads him to the completion state, $$S_2$$, and gets a reward $$R$$.

How do we form the target value $$Q_\text{target}$$ for $$S_1$$ now? Do we still include the $$Q_\text{future}$$ term? Or is it only the reward in this case? I am not sure if $$Q_\text{future}$$ even has meaning after reaching the final state $$S_2$$. So, I think that, for the final step, the target value must simply be the reward. Is this right?

• Although your question is put in the context of DQN and Q-learning, this could be closed as a duplicate of ai.stackexchange.com/q/3758/2444, which is asked in the context of SARSA: for this reason, I will actually not mark your post as a duplicate, although both questions have the same answer. Here and here are other related/similar/duplicate questions.
– nbro
Nov 1 '20 at 14:21

Now, let us assume the agent is in the penultimate state, $$S_1$$, and chooses the action $$A$$ that leads him to the completion state, $$S_2$$, and gets a reward $$R$$.

How do we form the target value $$Q_\text{target}$$ for $$S_1$$ now? Do we still include the $$Q_\text{future}$$ term? Or is it only the reward in this case?

Your term "completion state" is commonly called "terminal state". In a terminal state, there are no more actions to take, no more time steps, and no possibility to take any action. So, by definition, in your state $$S_2$$, the expected future reward is $$0$$.

Mathematically, this is often noted like $$v(S_T) = 0$$ or $$q(S_T,\cdot) = 0$$ with the $$T$$ standing for last time step of the episode, and dot standing in for the fact that no action need to be supplied, or the specific action value is not relevant. So, therefore using your terms, $$Q_\text{future} = \text{max}_a Q(S_2, a) = 0$$

That makes the equations work in theory, but does not explain what to do in code. In practice in your code, you would do as you suggest and use just the reward when calculating the TD target for $$Q(S_1, A)$$. This is typically done using an if block around the done condition e.g.

if done:
td_target = r
else:
td_target = r + gamma * np.max(q_future_values)
end


Of course, the details depend on how you have structured and named your variables. You will find code similar to this in most DQN implementations though.

You should not really try to learn $$V(S_2)$$ or $$Q(S_2, A)$$, or calculate TD target starting from $$S_2$$, because the result should be $$0$$ by definition.