What should the target be when the neural network outputs multiple Q values in deep Q-learning?

Question

I have some gaps in my understanding regarding the performing of the gradient descent in Deep - Q networks. The original deep q network for Atari performs a gradient descent step to minimise $y_j - Q(s_j,a_j,\theta)$, where $y_j = r_j + \gamma max_aQ(s',a',\theta)$.

In the example where I sample a single experience $(s_1,a_2,r_1,s_2)$ and I try to conduct a single gradient descent step, then feeding in $s_1$ to the neural network outputs an array of $Q(s_1,a_0), Q(s_1,a_1), Q(s_1,a_2), \dots$ values.

When doing gradient descent update for this single example, should the target output to set for the network be equivalent to $Q(s_1,a_0), Q(s_1,a_1), r_1 + \gamma max_{a'}Q(s_2,a',\theta), Q(s_1,a_3), \dots$ ?

I know the inputs to the neural network to be $s_j$, to give the corresponding Q values. However, I cannot concretize the target values that the network should be optimized.

Answer 1

When doing gradient descent update for this single example, should the target output to set for the network be equivalent to $Q(s_1,a_0), Q(s_1,a_1), r_2 + \gamma max_aQ(s',a',\theta) , Q(s_1,a_3),...$ ?

Other than what looks like a couple of small typos, then yes.

This is an implementation issue for DQN, where you have decided to create a function that outputs multiple Q functions at once. There is nothing about this in Q learning theory, so you need to figure out what will generate the correct error (and therefore gradients) for an update step.

You don't know the TD targets for actions that were not taken, and cannot make any update for them, so the gradients for these actions must be zero. One way to achieve that is to feed back the network's own output for those actions. This is common practice because you can use built-in functions from neural network libraries to handle minibatches*.

There are some details worth clarifying:

• You have substituted the third entry in the array with the calculated TD target because the action from experience replay is $a_2$. In general you substitute for the action taken. Looks like you have this correct.

• You have $r_1$ in your experience replay table, but put $r_2$ in your TD target formula. Looks like a typo. Another typo is that you maximise over $a$ but reference $a'$. Also, you reference $s'$ but don't define it anywhere. Fixing these issues gives $r_1 + \gamma \text{max}_{a'}Q(s_2,a',\theta)$

• For the TD target it is often worth using a dedicated target network that every N steps is copied from the learning network. It helps with stability. This can be noted as a "frozen copy" of $\theta$ noted $\theta^-$, and the neural network approximate Q function often noted $\hat{q}$ giving formula of $r_1 + \gamma \text{max}_{a'}\hat{q}(s_2,a',\theta^-)$ for your example.

---

* If you want you can also calculate the gradient more directly from the single action that was taken, and back propagate from there, knowing that all the other outputs will have a gradient componnet of zero. That requires implementing at least some of the back propagation yourself.

Answer 2

You are looking for the best actions which minimize the loss function. You sample a batch of memory buffer uniformly and define a loss function based on that batch. The memory buffer consists of trajectories. Each trajectory consists of an state and the action taken in that state which results in next state and an immediate reward. If the trajectory is shown by $(s,a,r,s\prime)$, the loss for this single state is simply defined as: $(r + max_a\prime Q(s\prime,a\prime,w^-)-Q(s,a,w))^2$.

The minus sign above the parameters means that you should fix the the target parameters to ensure the stability of learning. So the loss function for a whole batch is: $L(w) = E_{(s,a,r,s\prime)\sim U(D)}(r + max_a\prime Q(s\prime,a\prime,w^-)-Q(s,a,w))^2$.