What is the target Q-value in DQNs?

Question

I understand that in DQNs, the loss is measured by taking the MSE of outputted Q-values and target Q-values.

What does the target Q-values represent? And how is it obtained/calculated by the DQN?

Answer 1

When training a Deep Q network with experienced replay, you accumulate what is known as training experiences $e_t = (s_t, a_t, r_t, s_{t+1})$. You then sample a batch of such experiences and for each sample you do the following.

1. Feed $s_t$ into the network to get $Q(s,a;\theta)$.
2. Feed $s_{t+1}$ into the network to get $Q(s’,a’,\theta)$.
3. Choose $max_aQ(s’,a’,\theta)$ and set $ \gamma max_aQ(s’,a’,θ)$ + $r_t$ as the target of the network.
4. Train the network with $s_t$ as input to update $\theta$. The output from the input of $s_t$ is $Q(s,a,\theta)$ and the gradient descent step minimises the squared distance between $Q(s,a,\theta)$ and $\gamma max_aQ(s’,a’,θ)$ + $r_t$

Answer 2

The deep Q-learning (DQL) algorithm is really similar to the tabular Q-learning algorithm. I think that both algorithms are actually quite simple, at least, if you look at their pseudocode, which isn't longer than 10-20 lines.

Here's a screenshot of the pseudocode of DQL (from the original paper) that highlights the Q target.

Here's the screenshot of Q-learning (from Barto and Sutton's book) that highlights the Q target.

In both cases, the $\color{red}{\text{target}}$ is a reward plus a discounted maximum future Q value (apart from the exception of final states, in the case of DQL, where the target is just the reward).

There are at least 3 differences between these two algorithms.

• DQL uses gradient descent because the $Q$ function are represented by neural networks rather than tables, like in Q-learning, and so you have an explicit loss function (e.g. MSE).

• DQL typically uses experience replay (but, in principle, you could also do this in Q-learning)

• DQL encodes the states (i.e. $\phi$ encodes the states).

Apart from that, the logic of both algorithms is more or less the same, so, if you know Q-learning (and you should know it before diving into DQL), then it shouldn't be a problem to learn DQL (if you also have a decent knowledge of deep learning).

Answer 3

What does the target Q-values represent?

In a DQN, which uses off-policy learning, they represent a refined estimate for the expected future reward from taking an action $a$ in state $s$, and from that point on following a target policy. The target policy in Q learning is based on always taking the maximising action in each state, according to current estimates of value.

The estimate is refined in that it is based on at least a little bit of data from experience - the immediate reward, and what transition happened next - but generally it is not going to be perfect.

And how is it obtained/calculated by the DQN?

There are lots of ways to do this. The simplest in DQN is to process a single step lookahead based on the experience replay table.

If your table contains the tuple [state, action, immediate reward, next state, done?] as $[s, a, r, s', d]$ then the formula for TD target, $g_{t:t+1}$ is

$$r + \gamma \text{max}_{a'}[Q_{target}(s',a')], \qquad \text{when}\space d \space \text{is false}$$

$$r, \qquad \text{when}\space d \space \text{is true}$$

Typically $Q_{target}$ is calculated using the "target network" which is a copy of the learning network for Q that is updated every N steps. This delayed update of the target predictions is done for numerical stability in DQN - conceptually it is an estimate for the same action values that you are learning.

This target value can change every time you use any specific memory from experience replay. So you have to perform the same calculations on each minibatch, you cannot store the target values.