What is the target output for updating a Deep Q Network

Question

I'm trying to implement Deep Q-Learning for a pet problem having a continuous state space and discretized action space.

The algorithm for table-based Q-Learning updates a single entry of the Q table - i.e. a single $Q(s, a)$. However, a neural network outputs an entire row of the table - i.e. the Q-values for every possible action for a given state. So, what should the target output vector be for the network?

I've been trying to get it to work with something like the following:

q_values = model(state)
action = argmax(q_values)
next_state = env.step(state, action)
next_q_values = model(next_state)
max_next_q = max(next_q_values)

target_q_values = q_values
target_q_values[action] = reward(next_state) + gamma * max_next_q

The result is that my model tends to converge on some set of fixed values for every possible action - in other words, I get the same Q-values no matter what the input state is. (My guess is that this is because, since only 1 Q-value is updated, the training is teaching my model that most of its output is already fine.)

What should I be using for the target output vector for training? Should I calculate the target Q value for every action, instead of just one?

Answer 1

As you say, the output of a $Q$ network is typically a value for all actions of the given state. Let us call this output $\mathbf{x} \in \mathbb{R}^{|\mathcal{A}|}$. To train your network using the squared bellman error you need first calculate the scalar target $y = r(s, a) + \max_a Q(s', a)$. Then, to train the network we take a vector $\mathbf{x'} = \mathbf{x}$ and change the $a$th element of it to be equal to $y$, where $a$ is the action you took in state $s$; call this modified vector $\mathbf{x'}_a$. We calculate the loss $\mathcal{L}(\mathbf{x}, \mathbf{x'}_a)$ and back propagate through this to update the parameters of our network.

Note that when we use $Q$ to calculate $y$ we typically use some form of target network; this can be a copy of $Q$ where the parameters are only updated every $i$th update or a network whose weights are updated using a polyak average with the main networks weights after every update.

Judging by your code it looks as though your action selection is what might be causing you some problems. As far as I can tell you're always acting greedily with respect to your $Q$-function. You should be looking to act $\epsilon$-greedily, i.e. with probability $\epsilon$ take a random action and act greedily otherwise. Typically you start with $\epsilon=1$ and decay it each time a random action is taken down to some small value such as 0.05.

Answer 2

There are a couple ways you can define the architecture of a DQN. The most common way of doing it is by taking in the states and outputting the value function of all possible actions - this leads to a DQN with multiple outputs. The other, less efficient way, includes taking in an state-action as input and outputting a single real value - this approach is typically avoided since we need to run the model multiple times to get estimates for different actions.

The replay buffer is used to store $(S,A,R,S')$ transitions as encountered using your $\epsilon$-soft policy. We sample one of these transitions from the replay buffer and calculate an estimate of the value function for $(S,A)$ i.e $\hat Q(S,A,\theta)$ and then we calculate a target as follows. $$target =R+\max_\limits{a'}\hat Q(S',a',\theta^-)$$

Assuming you use the first model, you can then use a Squared error loss function, defined as follows, and modify your parameter as a function of that

$$L(\theta) = (target-\hat Q(S,A,\theta))^2$$

Assuming for now the target is fixed (I'll explain this in a minute), only $Q(S,A,\theta)$ is a function of $\theta$ in the loss function. $Q(S,A,\theta)$ corresponds to one output node of your DQN and therefore, as you've already highlighted, when carrying out EBP the parameters are updated such that we make the value of this one node tend to the specified target.

This is just how Q-learning works, we use samples generated by the behaviour policy to create $L(\theta)$ and then tweak the parameters to minimise the cost. As we do this for more and more samples the network hopefully figures out a way that accommodates for every sample it's been trained on so far (with more emphasis on the most recent samples).

As to your issue, are you sure you're training on multiple different samples and not just a specific one? it may just be a bug you've overseen.

---

Explaining $\theta^-$

I used a slightly different notation, $\theta^-$, for the parameters used to generate the bootstrapped estimate, $\max_\limits{a'}\hat Q(S',a,\theta^-)$. $\theta^-$ is only matched to $\theta$ every $n^{th}$ step because we want to keep the target constant as much as possible. The reason for this is because Q-learning does not necessarily converge when using neural networks partly due to bootstrapping which can cause a divergence of optimisation because of state generalisation. By using this $\theta^-$ we help prevent things like this from happening.

Ultimately the idea of the replay buffer and the fixed parameter for bootstrapping are to try to convert the RL problem into a supervised learning problem because we know much more about how to deal with supervised learning problems when using DNNs.